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As  you  requested,  this  report  assesses  whether  commercial  product  development  practices 
offer  ways  to  improve  the  process  used  at  the  Department  of  Defense  (dod)  to  prepare  weapon 
systems  for  production.  It  also  determines  how  differences  in  commercial  and  non 
environments  for  developing  new  products  affect  the  corresponding  practices.  We  make 
recommendations  to  the  Secretary  of  Defense  and  present  matters  for  congressional 
consideration,  all  of  which  are  intended  to  create  the  necessary  environment  to  ensure  dod’s 
future  success  in  improving  the  transition  of  weapons  from  development  to  production. 

We  are  sending  copies  of  this  report  to  other  congressional  committees;  the  Secretaries  of 
Defense,  the  Army,  the  Navy,  and  the  Air  Force;  and  the  Director,  Office  of  Management  and 
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questions.  Other  major  contributors  to  this  report  are  listed  in  appendix  11. 
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Executive  Summary 


Purpose 


The  Department  of  Defense  (dod)  continues  to  state  a  need  to  modernize 
weapons  for  the  armed  forces  at  a  faster  pace  within  relatively  level  funds. 
It  has  a  budget  of  over  $40  billion  in  fiscal  year  1998  to  acquire  and 
upgrade  weapons,  and  may  not  reasonably  expect  to  receive  much  more 
than  that  amount  in  future  years.  Therefore,  it  must  find  new  ways  to 
modernize  more  economically.  The  challenge  of  the  Soviet  Union  has  been 
replaced  by  a  challenge  to  maintain  weapons  that  have  a  leading 
technological  edge;  however,  such  technologies  are  increasingly  being 
developed  in  the  commercial  sector.  In  this  period  of  lessened  tensions, 
DOD  has  an  opportunity  to  revamp  the  practices  it  uses  to  acquire  weapon 
systems,  avoid  the  technological  obsolescence  that  comes  with  15-year 
and  longer  product  development  cycles,  speed  the  pace  of  modernization 
in  the  face  of  budgetary  pressures,  and  meet  defense  needs  with  sufficient 
flexibility.  The  practices  employed  by  some  commercial  firms  to  reduce 
the  time  and  money  to  develop  new  products — ^by  as  much  as 
50  percent — can  illuminate  ways  for  dod  to  mahe  similar  improvements. 

In  response  to  a  request  from  the  Chairman  and  Ranking  Minority 
Member,  Subcommittee  on  Acquisition  and  Technology,  Committee  on 
Armed  Services,  gao  assessed  whether  commercial  practices  offer  ways  to 
improve  dod’s  process  for  transitioning  weapons  from  development  to 
production.  Specifically,  this  report  (1)  compares  dod’s  practices  for 
preparing  a  weapon  system  for  production  with  best  commercial 
practices,  (2)  determines  how  differences  in  commercial  and  dod 
environments  for  developing  new  products  affect  practices,  and 
(3)  discusses  environmental  changes  that  are  key  to  the  success  of  dod 
initiatives  for  improving  the  transition  of  weapons  flrom  development  to 
production. 


Background 


Commercial  firms  make  a  distinction  between  product  and  technology 
development.  Product  development  entails  the  design  and  manufacture  of 
a  product,  such  as  an  airplane,  a  car,  or  a  satellite,  as  an  end  item  for 
delivery  to  a  customer.  The  basic  features  of  product  development  include 
a  definable  market  in  terms  of  customer  needs  or  wants,  the  ability  to 
design  a  product  for  that  market  opportunity,  the  ability  to  produce  that 
product,  and  the  investment  capital  needed  to  fund  the  development 
effort  Technology  development  fosters  technological  advances  for 
potential  application  to  a  product  development.  In  dod,  the  acquisition 
cycle  for  weapons  includes  technology  development,  product 
development,  and  production.  Thus,  the  distinction  between  product  and 
technology  development  on  an  individual  program  is  much  less  clear. 
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Among  the  key  sources  of  information  gao  relied  on  in  this  review  were 
individual  dod  acquisition  programs  and  several  firms  recognized  as  being 
among  the  best  in  developing  and  manufacturing  new  products.  Much 
commonality  existed  among  these  firms’  practices,  even  though  their 
products  spanned  a  range  of  technological  sophistication.  In  this  report, 
GAO  highlights  the  best  commercial  practices  in  product  development 
based  on  its  fieldwork.  As  such,  they  are  not  intended  to  describe  all 
commercial  industry  or  suggest  that  commercial  firms  are  without  flaws. 


Results  in  Brief 


Commercial  firms  gained  more  knowledge  about  a  product’s  technology, 
performance,  and  producibility  much  earlier  in  the  product  development 
process  than  dod.  Product  development  in  commercial  ventures  was  a 
clearly  defined  undertaking  for  which  firms  insisted  on  having  the 
technology  in  hand  to  meet  customer  requirements  before  starting.  Once 
underway,  these  firms  demanded — and  got — specific  knowledge  about  a 
new  product  before  production  began.  The  process  of  discovery — ^the 
accumulation  of  knowledge  and  the  elimination  of  unknowns — ^was 
completed  for  the  best  commercial  programs  well  ahead  of  production. 
Not  having  this  knowledge  when  demanded  constituted  a  risk  the  firms 
found  unacceptable.  Inunature  or  undeveloped  technology  could  not  meet 
these  demands  and  was  kept  out  of  commercial  product  development 
programs;  this  technology  was  managed  separately  until  it  could  meet  the 
demands  for  product  development. 

In  contrast,  dod  programs  allowed  more  technology  development  to 
continue  into  product  development.  Consequently,  the  programs 
proceeded  with  much  less  knowledge — and  thus  more  risk — about 
required  technologies,  design  capability,  and  producibility.  The  programs’ 
discovery  process  persisted  much  longer,  even  after  the  start  of 
production.  Turbulence  in  program  outcomes — in  the  form  of  production 
problems  and  associated  cost  and  schedule  increases — ^was  the 
predictable  consequence  as  the  transition  to  production  was  made. 
Although  DOD  accepted  more  unknowns  on  its  programs  than  commercial 
firms,  it  understated  the  risks  present. 

The  commercial  and  defense  environments  created  different  incentives 
and  elicited  different  behaviors  firom  the  people  managing  the  programs. 
Specific  practices  took  root  and  were  sustained  because  they  helped  a 
program  succeed  in  its  environment — not  because  they  were  textbook 
solutions.  'The  success  of  commercial  product  developments  was 
determined  when  production  items  were  sold.  Until  that  point,  the  firm’s 


Page  3 


GAO/NSIAD- 98-56  Best  Practices 


Executive  Summary 


own  money  was  at  risk.  Failure  was  seen  as  both  clear  and  painful  if  the 
customer  walked  away.  This  definition  of  success,  coupled  with  the 
realization  that  production  startup  was  relatively  close  at  hand,  made 
production  concerns  a  key  factor  in  program  start  and  subsequent 
decisions  and  provided  strong  incentives  to  identify  risks  both  early  and 
realistically,  dod  programs  began  without  needed  technology  in  hand; 
rather,  they  were  encouraged  to  include  undeveloped  technology.  Because 
these  programs  ran  much  longer,  production  concerns  did  not  play  as  big  a 
role  and  were  not  as  critical  to  success  in  the  early  stages.  The  definition 
of  success  was  more  complicated  in  dod.  During  most  of  product 
development,  success  was  defined  as  getting  dod  and  the  Congress,  as  the 
customers,  to  fund  the  development  annually.  Optimistic  assessments  of 
performance  and  cost  helped  ensure  this  kind  of  success;  realistic  risk 
assessments  did  not 

Commercial  practices  for  gaining  knowledge  and  assessing  risks  can  help 
produce  better  outcomes  on  dod  acquisitions.  Indeed,  dod  has  several 
commercial-Uke  initiatives  underway,  such  as  using  cost  or  price  as  a 
means  for  forcing  technology  tradeoffs,  that  it  believes  are  having 
promising  results.  For  such  practices  to  work  on  a  broad  scale,  however, 
the  DOD  environment  must  be  conducive  to  such  practices.  At  least  two 
factors  are  critical  to  fostering  such  an  environment.  First,  program  launch 
decisions  must  be  relieved  of  the  need  to  overpromise  on  technical 
performance  and  resource  estimates.  The  pressure  to  amass  broad  support 
at  launch  creates  incentives  for  new  programs  to  embrace  far  more 
technology  development  than  commercial  programs.  Although  technology 
development  clearly  must  be  undertaken  by  dod  and  supported  by  the 
Congress,  its  objectives,  as  well  as  what  can  be  demanded  of  its 
knowledge  and  estimates,  differ  from  those  of  product  development. 
Second,  once  a  program  is  imderway,  it  must  become  acceptable  for 
program  managers  to  identify  unknowns  as  high  risks  so  that  they  can  be 
aggressively  worked  on  earlier  in  development.  Currently,  identifying  a 
high  risk  on  a  dod  program  is  perceived  as  inviting  criticism  and  the  loss  of 
funding.  Studies  sponsored  by  dod  and  the  defense  industry  call  for  the 
kinds  of  changes  that  could  help  shape  a  better  environment  for  managing 
weapon  acquisitions.  The  challenge  for  dod  and  the  Congress  is  to  foster 
the  environment  that  provides  program  managers  with  incentives  for 
applying  best  practices. 
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Principal  Findings 


Leading  Commercial  Firms 
Attain  More  Knowledge 
Earlier  in  Product 
Development 


To  minimize  the  amount  of  technology  development  that  occurs  during 
product  development,  the  companies  gao  visited  employ  a  disciplined 
process  to  match  requirements  with  technological  capability  before  the 
product  development  process  begins.  This  process  is  grounded  in 
production  realities  that  demand  proof  that  the  technology  will  work  and 
can  be  produced  at  an  acceptable  cost,  on  schedule,  and  with  high  quality. 
The  companies  bring  proven  technological  knowledge  to  the  requirements 
process  in  the  form  of  current,  high  fidelity  information  fi*om  predecessor 
programs,  people  with  first-hand  experience  on  those  programs,  and  new 
technologies  deemed  mature  as  a  result  of  having  “graduated”  from  a 
disciplined  technology  development  and  screening  process.  Also,  they 
communicate  extensively  with  customers  to  match  their  wants  and  needs 
to  the  firms’  available  technology  and  ability  to  manufacture  the  desired 
product.  They  do  not  stray  far  from  their  technological  foundation. 


For  the  programs  gao  reviewed,  dod  did  not  get  as  good  a  match  between 
what  customers  require  and  what  technology  could  confidently  deliver. 
Even  though  dod  examines  the  potential  for  available  technology  to  meet 
product  requirements,  it  allows  requirements  to  make  technology  reach 
beyond  what  is  proven,  dod  has  less  knowledge  about  the  ability  of  the 
design  to  deliver  required  performance  and  be  produced  at  this  point  in 
the  program  than  commercial  firms.  The  knowledge  that  the  match  has 
been  achieved  between  requirements  and  technology  is  often  not  attained 
until  testing  is  completed,  late  in  development  or  after  production  begins. 
Commercial  and  dod  decisions  made  on  a  lightweight  aircraft 
material — ^aluminum  lithium — ^are  illustrative  of  the  differences  between 
the  two  sectors.  Boeing  had  initially  decided  to  use  the  alloy  on  its  777-200 
aircraft  but  rejected  its  use  early  in  development  because  it  was  expensive, 
its  manufacturing  processes  were  not  well  understood,  and  its  availability 
was  limited,  dod  accepted  these  risks  and  used  the  alloy  on  the  C-17 
aircraft.  The  alloy  is  now  being  phased  out  of  the  program  as  some  of  its 
unknowns  became  actual  problems. 


Before  the  halfway  point  of  product  development,  the  commercial  firms 
GAO  visited  achieved  near  certainty  that  their  product  designs  would  meet 
customer  requirements  and  had  gone  a  long  way  to  ensure  that  the 
product  could  be  produced.  Both  dod  and  commercial  firms  hold  a  critical 
design  review  to  determine  whether  the  design  is  mature  and  essentially 
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complete  for  production  purposes.  Engineering  drawings,  which 
document  the  schematics  of  the  product,  along  with  its  performance, 
required  materials  and  production  processes,  comprise  much  of  the 
source  material  for  the  critical  design  review.  Commercial  firms  typically 
had  over  90  percent  of  these  drawings  available  for  the  review.  In 
comparison,  the  C-17  program  had  less  than  60  percent  and  the  F-22 
program  less  than  one-third  of  the  drawings  available  for  the  review.  Over 
one-fifth  of  the  C-17’s  drawings  became  available  after  production  began, 
and  the  aircraft  experienced  a  number  of  problems  in  production  as 
difficulties  with  the  design  were  worked  out.  Several  key  technologies  are 
still  unproven  on  the  F-22,  and  some  will  not  be  proven  until  after  40 
aircraft  have  entered  production.  Nonetheless,  the  risks  of  proceeding 
with  the  rest  of  development  as  planned  at  the  time  of  the  critical  design 
reviews  for  both  programs  were  deemed  acceptable. 

The  companies  gao  visited  reached  the  point  at  which  they  knew  that 
manufacturing  processes  would  produce  a  new  product  conforming  to 
cost,  quality,  and  schedule  targets  before  they  began  fabricating 
production  articles.  Reaching  this  point  meant  more  than  knowing  the 
product  could  be  manufactured;  it  meant  that  all  key  processes  were 
under  control,  such  that  the  qualify,  volume,  and  cost  of  their  output  were 
proven  and  acceptable.  The  dod  programs  demanded  less  proof  of  a 
design’s  producibility.  The  C-17  program  began  production  in  1989  and  still 
has  less  than  13  percent  of  its  key  manufacturing  processes  in  control.  The 
F-22  program  is  currently  faring  better  than  the  G-17:  the  contractor 
believes  it  has  almost  40  percent  of  its  key  manufacturing  processes  in 
control,  2  years  before  production  is  scheduled  to  begin.  However,  the 
program  does  not  plan  to  have  all  key  processes  in  control  until  about 
4  years  into  production. 


Differences  in  Military  and 
Commercial  Practices 
Reflect  Different 
Environments 


The  commercial  firms  gao  contacted  launch  a  product  development  only 
when  a  solid  business  case  can  be  made.  The  business  case  basically 
revolves  aroimd  the  ability  to  produce  a  product  with  the  right  features  to 
meet  the  market  opportunity  on  schedule,  with  limited  investment  capital, 
and  at  a  predictable  unit  cost  so  that  the  product  will  sell  well  enough  to 
make  an  acceptable  return  on  investment.  Because  success  is  determined 
in  production,  the  business  case  for  launching  a  program  considers 
production  realities  and  builds  in  natural  curbs  to  overreaching  for 
performance,  cost,  or  schedule.  A  company  demands  considerable  proof 
that  the  product  wDl  fulfill  all  of  the  business  case  factors  and  then 
provides  full  support  for  the  program  to  succeed.  The  business  case 
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provides  a  very  solid  decision-making  framework  from  the  outset  and 
throughout  the  program.  Commercial  companies  build  relatively  short 
cycle  times,  keyed  to  meeting  market  demands,  into  their  decisions  to 
begin  a  product’s  development.  These  short  timeframes,  together  with  the 
responsibility  for  protecting  the  business  case,  encourage  program 
managers  to  identify  risks  and  enable  them  to  say  “no”  to  pressures  to 
accept  unknowns.  The  companies  are  conservative  in  their  estimates  and 
aggressive  in  reducing  risk.  The  abundance  of  reliable  data  and 
experienced  people  from  predecessor  programs  provides  a  solid  factual 
basis  for  defining  unknowns  and  assessing  risks. 

Because  dod  launches  programs  earlier,  the  knowledge  commercial  firms 
insist  on  is  generally  not  available  for  a  dod  program  until  years  later. 

Some  knowledge,  such  as  high-fidelity  information  from  numerous  closely 
related  predecessor  programs,  is  simply  imavailable.  Even  though  less 
information  about  a  new  dod  product  development  is  available  at  the  time 
of  launch,  the  competition  for  funding  requires  detailed  projections  to  be 
made  from  the  information  that  does  exist.  Although  dod  is  attempting  to 
ease  the  technical  requirements  of  programs,  anew  product  development 
must  possess  performance  features  that  distinguish  it  from  either  the 
systems  to  be  replaced  or  rival  candidates.  Consequently,  aspiring  dod 
programs  are  encouraged  to  include  performance  features  and  design 
characteristics  that  rely  on  immature  technologies.  Untempered  by 
knowledge  to  the  contrary,  the  risks  associated  with  these  technologies 
are  deemed  acceptable.  Production  realities,  critical  to  matching 
technological  capabilities  with  customer  requirements  on  commercial 
programs,  are  too  far  away  from  the  dod  launch  decision  to  have  the  same 
curbing  effect  on  technology  decisions. 

In  this  environment,  risks  in  the  form  of  ambitious  technology 
advancements  and  tight  cost  and  schedule  estimates  are  accepted  as 
necessary  for  a  successful  launch.  Problems  or  indications  that  the 
estimates  are  decaying  do  not  help  sustain  the  program  in  subsequent 
years,  and  thus  their  admission  is  implicitly  discouraged.  An  optimistic 
production  cost  estimate  makes  it  easier  to  launch  a  product  development 
and  sustain  annual  approval;  admission  that  costs  are  likely  to  be  higher 
could  invite  failure.  There  are  few  rewards  for  discovering  and  recognizing 
potential  problems  early  in  the  dod  product  development,  given  the 
amount  of  external  scrutiny  the  programs  receive.  The  behavior  of 
tolerating  unknowns  and  not  designating  them  the  same  risk  value  as  in 
the  commercial  environment  is  manifested  in  the  dod  environment 
because  there  is  little  incentive  to  admit  to  high  risks  before  it  is 
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absolutely  necessaiy.  This  is  not  to  say  that  dod  program  managers  and 
other  sponsors  act  irrationally  or  with  bad  intentions.  Rather,  they  see  the 
programs  under  their  purview  as  aligned  with  the  national  interest  and  do 
what  they  believe  is  right,  given  the  pressures  they  face. 


The  Right  Environment 
Will  Be  Key  to  the  Success 
of  DOD  Initiatives  to 
Improve  Weapon 
Acquisitions 


DOD  has  embarked  on  several  acquisition  reform  initiatives  that  draw 
lessons  from  commercial  practices.  These  initiatives  include  using  cost  or 
price  targets  as  vehicles  for  forcing  requirements  or  technology  tradeoffs, 
evaluating  contractors’  past  performance  as  a  factor  in  selecting  sources 
for  new  contracts,  removing  specifications  that  told  contractors  how  to 
design  or  build  a  product  in  favor  of  specifying  what  performance  was 
required  of  the  end  item,  and  using  multidisciplinary  teams  to  make 
decisions  and  tradeoffs  on  individual  weapon  systems,  dod  reports  that 
programs  are  showing  potential  reductions  in  cycle  time  and  staff 
resources  as  a  result  of  these  initiatives,  dod  has  also  recently  set  up  a 
funding  reserve  to  offset  unexpected  cost  growth  to  mitigate  the  effect  of 
unknowns  on  programs.  In  past  acquisitions,  changing  the  mechanics  of  a 
we£q)on’s  development,  without  changing  aspects  of  its  environment  that 
determine  its  incentives,  did  not  produce  desired  results.  Nonetheless, 
dod’s  current  initiatives  could  help  the  transition  of  weapons  into 
production  if  the  environment  for  launching  programs  and  appraising  risks 
can  be  changed  to  better  approximate  the  commercial  environment. 


Studies  sponsored  by  dod  and  the  defense  industry  call  for  changes  that 
could  help  shape  such  an  environment.  A  May  1996  study  by  the  Defense 
Science  Board  recommended  replacing  the  current  weapon  acquisition 
process  with  one  that  emphasizes  incremental  technology  advancement, 
coupled  with  much  shorter  product  development  cycle  times.  The  study 
also  noted  that  technology  development  should  be  aggressively  pursued 
outside  the  realm  of  weapon  system  development  programs.  In  April  1996, 
the  National  Center  for  Advanced  Technologies  also  proposed  a  change  in 
the  DOD  weapons  acqmsition  process  to  reduce  cycle  time  to  the  3-  to 
5-year  range  by  following  an  incremental  technology  advancement 
approach.  'The  report  noted  that  although  concepts  such  as  dod’s 
initiatives  are  constructive,  there  is  no  reason  to  assume  that  these 
concepts  should  succeed  when  previous  good  ideas  did  not. 

A  December  1994  study  by  the  Defense  Systems  Management  College 
made  several  recommendations  to  create  an  environment  for  weapon 
systems  development  to  encourage  realism  in  reporting  program  status.  A 
significant  recommendation  is  that  the  individual  military  services  transfer 
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control  of  their  acquisition  organizations  and  people  to  the  Under 
Secretary  of  Defense  for  Acquisition  and  Technology.  The  study  noted  that 
the  Under  Secretary  would  then  be  empowered  to  reward  candor  and 
realism  in  reporting  through  the  use  of  assignments  and  promotions. 


Recommendations 


GAO  makes  several  recommendations  to  the  Secretary  of  Defense.  These 
recommendations  are  intended  to  (1)  ensure  that  sound  standards  for  the 
timing  and  quality  of  production-related  knowledge  are  applied  to 
individual  programs,  (2)  separate  technology  development  from  weapon 
system  programs  to  enable  dod  to  meet  higher  knowledge  standards  on 
those  programs,  and  (3)  use  decisions  on  individual  weapon  systems  to 
foster  the  right  environment  for  identifying  risks  early  and  send  the  right 
message  to  other  programs  concerning  what  practices  will  work  in  that 
environment.  These  recommendations  appear  in  full  in  chapter  5. 


Matters  for 

Congressional 

Consideration 


Because  of  its  critical  role  in  creating  the  environment  for  what 
constitutes  program  success  and  which  practices  will  work,  the  Congress 
may  wish  to  consider  supporting  the  Secretary  of  Defense’s  efforts  to 
create  such  an  environment  through  changes  to  the  acquisition  process 
that  provide  program  managers  clear  incentives  for  gaining  sufficient 
knowledge  at  key  points  in  weapon  acquisition  programs.  The  best 
commercial  practices  described  in  this  report  suggest  what  may  constitute 
“sufficient”  levels  of  knowledge.  If  dod  takes  steps  to  manage  technology 
development  efforts  outside  the  bounds  of  individual  weeqjon  system 
programs,  the  Congress  may  wish  to  consider  providing  the  funds  needed 
for  such  efforts.  The  Congress  could  also  help  create  the  right  incentives 
on  individual  programs  by  favorably  considering  dod  funding  requests  to 
mitigate  high  risks  early  in  a  program  and  cautiously  considering  late 
requests  for  funds  to  resolve  problems  that  should  have  been  addressed 
earlier. 


Agency  Comments 


DOD  agreed  with  the  report  and  all  of  the  recommendations.  A  discussion 
of  dod’s  actions  appears  on  pages  75-77.  dod’s  comments  appear  in 
appendix  I. 
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Introduction 


The  Department  of  Defense  (dod)  continues  to  state  a  need  to  modernize 
weapons  for  the  armed  forces  at  a  faster  pace  within  relatively  level  funds. 
It  has  a  budget  of  over  $40  billion  in  fiscal  year  1998  to  acquire  and 
upgrade  weapons,  and  may  not  reasonably  expect  to  receive  much  more 
than  that  amount  in  future  years.  Therefore,  it  must  find  new  ways  to 
modernize  more  economically.  The  challenge  of  the  Soviet  Union  has  been 
replaced  by  a  challenge  to  maintain  weapons  that  have  a  leading 
technological  edge;  however,  such  technologies  are  increasingly  being 
developed  in  the  commercial  sector.  In  this  period  of  lessened  tensions, 
DOD  has  an  opportunity  to  revamp  the  practices  it  uses  to  acquire  weapon 
systems,  avoid  the  technological  obsolescence  that  comes  with  16-year 
and  longer  product  development  cycles,  speed  the  pace  of  modernization 
in  the  face  of  budgetary  pressures,  and  meet  defense  needs  with  sufficient 
flexibility.  When  dealing  with  some  similar  problems,  the  best  commercial 
firms  have  focused  on  the  early  stages  of  the  product  development  cycle  to 
speed  up  product  developments,  get  them  into  production  sooner,  and 
achieve  cost  goals. 


Product  Development 
in  the  Commercial 
and  Defense  Sectors 


For  commercial  firms,  developing  a  product  entails  the  design  and 
manufacture  of  an  end  item,  such  as  an  airplane,  car,  or  satellite,  for 
delivery  to  a  customer.  The  basic  features  of  product  development  include 
a  definable  market  in  terms  of  customer  needs  or  wants,  the  ability  to 
design  a  product  for  that  market  opportunity,  the  ability  to  produce  that 
product,  and  the  investment  capital  needed  to  fund  development.  Product 
development  is  therefore  very  concerned  with  production  realities  and 
cost  and  schedule  targets.  It  is  distinct  from  technology  development, 
which  furthers  the  advancement  of  technology  to  see  if  it  has  potential 
application  to  a  product.  For  example,  a  firm  that  makes  satellites  may 
study  new  materials  and  their  properties  to  see  if  they  can  reduce  the 
weight  and  size  of  solar  array  batteries,  thus  improving  a  satellite’s 
carrying  capacity.  If  the  technology  can  be  perfected  and  is  feasible  to  use, 
it  is  ready  to  be  included  in  a  new  product  development.  More  failures  are 
expected  in  a  technology  development,  since  it  is  a  process  of  discovery 
that  deals  with  many  unknowns. 


In  DOD,  the  same  functions  take  place  on  the  weapon  systems  dod 
develops,  but  they  occur  within  the  broader  context  of  the  acquisition 
cycle.  The  entire  cycle  includes  technology  development,  product 
development,  and  production,  dod’s  development  process  is  designed  to 
manage  a  program  through  sequential  phases,  all  followed  by  major 
milestone  decisions  in  which  decisionmakers  approve  or  disapprove  the 
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acquisition  strategy  for  a  program  and  its  move  into  the  next  phase  based 
on  progress  made  as  presented  by  the  program  manager.  The  phases  of  the 
acquisition  cycle,  as  shown  in  figure  1.1,  are:  concept  exploration;  program 
definition  and  risk  reduction;  engineering  and  manufacturing  development 
(emd);  and  production,  fielding/deployment,  and  operational  support.  The 
concept  exploration  phase  decides  what  kind  of  weapon,  if  any,  is  the  best 
solution  to  a  military  need.  A  new  program  actually  starts  with  the 
program  definition  and  risk  reduction  phase. 
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Figure  1.1:  POP’s  Acquisition  Process 


Source:  DOD. 
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Within  the  acquisition  cycle  for  a  given  weapon  system,  technology 
development,  product  development,  and  production  activities  take  place 
over  a  number  of  years.  The  delineation  of  these  activities  is  not  always 
clearly  discemable.  Thus,  even  though  the  product  development  activities 
described  for  the  commercial  sector  would  occur  largely  in  a  weapon’s 
EMD  phase,  technology  development  and  production  activities  also  occur 
in  this  phase.  Likewise,  some  product  development  activities  for  a  weapon 
system  take  place  during  production.  The  acquisition  process  is  embodied 
in  dod’s  Regulation  5000.2,  Mandatory  Procedures  for  Major  Defense 
Acquisition  Programs,  which  was  revised  in  1996.  In  addition,  dod’s  1985 
Manual  4245.7-M,  Transition  From  Development  to  Production,  contains 
risk-reduction  guidance  based  on  best  practices. 

At  any  given  time,  dod  has  hundreds  of  products  in  various  stages  of 
development  and  production,  each  with  a  program  manager  from  one  of 
the  three  services  or  other  procuring  agencies  within  dod.  Although  dod 
maintains  oversight  for  all  of  these  acquisitions  and  controls  their  funding, 
the  actual  management  practices  employed  are  specific  to  the  program, 
the  contractors,  and  the  relevant  government  organization.  We  have  issued 
a  number  of  reports  on  weapon  system  acquisitions  since  the  1970s. 

The  program  manager  and  staff  for  a  new  product’s  development  in  either 
sector  face  a  formidable  task.  They  are  responsible  for  balancing  the 
requirements  for  the  product’s  performance  against  its  established  cost 
and  schedule  targets.  This  responsibility  includes  preparing  an  acquisition 
strategy  for  funding,  testing,  and  manufacturing  the  product  and  ensuring 
that  the  strategy  is  carried  out.  Throughout  the  development  process,  the 
program  manager  is  expected  to  identify  and  manage  high-risk  areas, 
which  means  making  decisions  about  tradeoffs  among  cost,  schedule,  and 
performance  to  deliver  a  product  that  will  satisfy  the  customer.  The 
program  manager  and  staff  must  therefore  have  expertise  in  many  areas, 
such  as  estimating  techniques,  the  budget  process,  simulation,  testing, 
engineering,  logistics,  and  production.  The  program  manager  is  the  one 
person  who  ultimately  must  know  whether  there  is  a  match  among 
financial,  time,  and  personnel  resources  provided  by  upper  management 
and  what  the  product  development  effort  can  deliver.  In  product 
development,  the  financial  investment  is  substantial  and  the  stakes  are 
high;  program  failure  is  not  taken  lightly. 
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The  Case  for  a 
Smoother  and  Faster 
Development  to 
Production  Cycle 


Products  that  proceed  more  quickly  and  smoothly  through  development 
into  production  require  lower  financial  investment  costs  to  develop.  Also, 
a  smoother  and  faster  process  enhances  the  product’s  impact  on 
customers  or  users  and  benefits  the  producer  by  providing  improved 
profitability  and  market  share.  Commercial  firms  that  have  been 
recognized  as  industry  leaders  have  significantly  reduced  the  time  it  takes 
to  complete  the  product  development  cycle.  They  have  put  new  products 
into  production  more  quickly  within  unit  production  cost  targets  and 
improved  the  products’  performance  features  and  quality.  For  example, 
Boeing  will  reduce  development  time  by  40  percent  on  its  777-300  airplane 
compared  with  its  777-200  development,  Hughes  develops  satellites  in 
26  months  and  will  soon  reduce  the  time  to  12  months,  and  Chrysler  has 
reduced  its  new  vehicle  development  times  from  over  5  years  to  less  than 
30  months. 


If  DOD  made  reductions  of  the  same  magnitude  in  the  time  it  takes  to 
develop  a  weapon,  it  could  reap  similar  benefits.  A  compelling  reason  for 
DOD  to  pursue  such  reductions  is  to  resolve  long-standing  difficulties  in 
controlling  cost  and  schedule  outcomes  of  major  weapon  system 
programs.  According  to  cost  and  schedule  data  firom  93  major  acquisitions 
started  since  1976,  the  acquisitions  overran  original  schedule  estimates  by 
an  average  of  24  percent.  Recent  dod  studies  have  concluded  that  weapon 
system  development  programs  typically  overrun  costs  by  20  to  40  percent 
and  that  acquisitions  average  16  to  18  years.  A 1993  Rand  study  concluded 
that  there  had  been  no  improvement  in  controlling  cost  growth  on  the 
average  weapon  system.  These  results  have  persisted  despite  the 
implementation  of  various  initiatives  to  mitigate  cost  risk  and  growth, 
including  significant  risk  management  guidelines  dod  instituted  in  1985  to 
improve  the  transition  to  production. 


Another  reason  for  dod  pursuing  faster  and  smoother  development  to 
production  cycles  is  to  lower  its  investment  costs,  enabling  the  services  to 
modernize  at  a  faster  pace  within  existing  funding  levels.  According  to 
DOD,  modernization  in  recent  years  has  been  sacrificed  to  improve 
readiness  in  the  forces,  dod  believes  that  modernization  should  proceed 
more  quickly  and  wants  to  increase  the  annual  investment  in  procurement 
by  $20  billion,  but  such  increases  have  not  materialized.  Shorter 
development  cycles  would  also  help  mitigate  the  possibility  that 
technology  will  become  obsolete  while  a  weapon  is  stiU  in  development. 
Technological  obsolescence  has  become  a  significant  challenge  for  dod 
acquisition  programs  whose  developments  span  many  years.  Finally,  faster 
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product  developments  would  deliver  improved  products  and  better 
capabilities  to  the  military  forces  sooner. 

DOD  recognizes  the  importance  of  reducing  product  development  costs  and 
cycle  times,  dod’s  1996  Technology  Area  Plan  recognizes  that  the 
prevailing  acquisition  environment  made  affordability  and  rapid  cycle 
times  for  acquisition  of  new  systems  key  to  maintaining  an  appropriate 
mix  of  systems  and  forces  that  are  ready  to  respond  to  the  defense 
missions  of  the  future.  The  plan  recognized  that  short  cycle  times 
invariably  create  cost  reductions  that  would  have  clear  implications  for 
readiness  and  modernization.  It  also  noted  that  world-class  commercial 
companies  have  demonstrated  overall  product  development  and 
production  cost  reductions  as  well  as  cycle  time  reductions  of  about 
50  percent.  The  plan  states  that,  despite  the  major  differences  in  products 
and  requirements,  cost  and  cycle  time  savings  in  that  same  range  are 
feasible  for  miUtary  programs  as  well. 

DOD  is  attempting  to  improve  the  product  development  process  by 
introducing  acquisition  reform  initiatives  that  mirror  world-class 
companies’  product  development  practices,  dod  believes  that  these 
initiatives  will  result  in  better  outcomes.  A  recent  Defense  Science  Board 
study^  concluded  that  the  research  and  development  phase  of  military 
systems  should  be  revamped  and  should  adopt  best  cormnercial  practices. 
The  aerospace  industry  has  also  offered  suggestions  based  on  commercial 
practices  for  improving  dod’s  development  process  for  weapons  as  a  basis 
for  change. 


Objectives,  Scope, 
and  Methodology 


The  Chairman  and  the  Ranking  Minority  Member,  Subcommittee  on 
Acquisition  and  Technology,  Senate  Committee  on  Armed  Services, 
requested  that  we  examine  various  aspects  of  the  acquisition  process  to 
identify  best  practices  that  might  be  useful  or  should  be  strengthened.  This 
report  covers  one  aspect  of  the  acquisition  process,  the  transition  from 
development  to  production.  Our  overall  objective  was  to  determine 
whether  commercial  practices  offer  ways  to  improve  dod’s  product 
development  process  as  it  relates  to  the  transition  to  production. 
Specifically,  this  report  (1)  compares  dod’s  practices  for  preparing  a 
weapon  system  for  production  with  best  commercial  practices, 

(2)  determines  how  differences  in  the  conunercial  and  dod  environments 
for  developing  new  products  affect  specific  practices,  and  (3)  discusses 


Streamlined  Approach  to  Weapon  Systems  Research,  Development  and  Acquisition:  The 
Application  of  Commercial  Practices,  a  Report  from  the  Defense  Science  Board  Task  Force  on 
Defense  Acquisition  Reform,  May  1996. 
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environmental  changes  that  are  key  to  the  success  of  dod  initiatives  for 
improving  the  transition  of  weapons  from  development  to  production. 

Even  though  we  selected  firms  with  product  lines  of  varying  complexity, 
we  did  not  concentrate  only  on  firms  whose  products  had  the  most  in 
common  with  weapon  systems.  Such  an  approach  would  have  limited  our 
ability  to  include  firms  recognized  as  the  best  at  getting  new  products  into 
production.  In  our  analysis,  we  concentrated  on  the  criteria  and 
knowledge  used  to  support  product  decisions.  Although  the  approach 
firom  product  to  product  may  vary,  the  basic  processes  and  standards  the 
best  commercial  firms  applied  to  product  decisions  were  consistent.  We 
were  limited  in  our  ability  to  obtain  and  present  some  relevant  data  that 
commercial  companies  considered  proprietary  in  nature.  This  information 
included  funding  amounts  for  investing  in  product  development  or  for 
recurring  production  costs;  the  exact  number  and  cost  of  changes  to 
engineering  drawings  on  a  product;  and  specific  costs  of  scrap,  rework, 
and  repair  on  products. 

To  determine  historical  cost  and  schedule  outcomes  on  past  dod 
acquisitions,  we  examined  Rand’s  database  of  Selected  Acquisition 
Reports  for  over  200  acquisitions  since  the  1960s.  To  determine  current 
DOD  policy  and  practices  and  identify  new  initiatives,  we  interviewed  and 
obtained  documents  from  officials  of  the  Office  of  the  Secretary  of 
Defense  in  Washington,  D.C.  For  information  from  acquisitions  on  specific 
risk  management  practices  and  results,  we  interviewed  officials  and 
reviewed  acquisition  and  risk  management  documentation  from  the 
following  two  major  dod  acquisition  programs,  which  are  in  different 
stages  of  the  process: 

The  C-17  aircraft,  an  air  refuelable,  four-engine  jet  transport,  is  being 
developed  by  McDonnell  Douglas  Corporation  (since  acquired  by  Boeing). 
It  received  full-rate  production  authority  in  November  1995.  The  Air  Force 
estimates  the  cost  to  develop  and  produce  120  C-17  aircraft  at  about  $43 
billion,  amoimting  to  a  program  unit  cost  of  $358  million.  Procurement 
unit  cost  is  estimated  by  the  Air  Force  at  $298  million. 

The  F-22  fighter,  the  next-generation  air  superiority  fighter,  is  being 
developed  by  Lockheed  Martin  Aeronautical  Systems  Company.  It  is 
currently  in  emd.  The  Air  Force  estimates  the  cost  to  develop  and  produce 
339  F-22s  at  about  $62.2  billion,  amounting  to  a  program  unit  cost  of 
$183  million.  Procurement  unit  cost  is  estimated  at  $127  million. 
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We  also  reviewed  the  following  newer  dod  acquisition  programs  that  have 
implemented  some  of  dod’s  acquisition  reform  initiatives;  because  these 
programs  are  newer,  our  review  was  more  limited: 

•  The  AIM-9X  Sidewinder  Missile,  a  joint  Navy/Air  Force,  launch-and-leave  air 
intercept  missile,  is  being  developed  by  Hughes  Corporation.  The  program 
is  currently  in  emd.  The  services  estimate  the  cost  to  develop  and  produce 
10,049  AIM-9X  missiles  at  about  $3.2  billion,  amounting  to  a  program  unit 
cost  of  $320,000.  Procurement  unit  cost  is  estimated  at  $264,000. 

•  The  Joint  Direct  Attack  Munitions  (jdam),  a  j  oint  Navy/Air  Force  tail  kit  to 
be  attached  to  a  2,000-pound  free-fall  bomb  and  converting  it  to  a  guided 
munition,  is  being  developed  by  McDonnell  Douglas  Corporation  (since 
acquired  by  Boeing).  The  jdam  program  is  currently  in  emd.  The  services 
estimate  the  cost  to  develop  and  produce  87,500  jdam  kits  at  about 
$3.39  billion,  amounting  to  a  program  unit  cost  of  $38,700.  Procurement 
unit  cost  is  estimated  at  $32,900. 

To  understand  the  product  development  process  and  best  practices  from 
the  commercial  sector,  we  conducted  general  literature  searches  and 
focused  those  searches  as  the  assignment  progressed.  We  also  met  with 
several  experts  in  the  area  of  product  development,  including 
representatives  of  Indiana  University,  Bloomington,  Indiana;  the  University 
of  Michigan,  Ann  Arbor,  Michigan;  and  the  Navy’s  Center  of  Excellence  for 
Best  Manufacturing  Practices,  College  Park,  Maryland.  On  the  basis  of  our 
literature  searches  and  discussions  with  experts,  we  identified  a  number 
of  commercial  firms  as  having  innovative  development  processes  and 
practices  that  resulted  in  successful  transitions  to  production.  We 
developed  a  data  collection  instrument  to  assist  us  in  gathering  uniform, 
quantifiable  measurements  about  each  firm’s  product  development 
process  and  practices  and  the  results  they  accomplished.  We  visited  the 
following  commercial  firms,  all  identified  in  our  literature  searches,  and 
followed  the  same  agenda  with  each  one: 

•  Boeing  Commercial  Airplane  Group  (airplane  manufacturer),  Everett, 
Washington; 

•  Chrysler  Corporation  (automobile  manufacturer).  Auburn  Hills,  Michigan; 

•  Cummins  Engine  Company  (engine  manufacturer),  Columbus,  Indiana; 

•  Ford  Motor  Company  (automobile  manufacturer).  Dearborn,  Michigan; 

•  Honda  Motor  Company  (automobile  manufacturer),  Raymond,  Ohio;  and 

•  Hughes  Space  and  Communications  (satellite  and  spacecraft 
manufacturer),  Los  Angeles,  California 
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Our  report  highlights  the  best  commercial  practices  in  product 
development  based  on  our  fieldwork.  As  such,  they  are  not  intended  to 
describe  all  commercial  industry  and  practices  or  suggest  that  commercial 
firms  are  without  flaws. 

We  conducted  our  review  between  October  1996  and  September  1997  in 
accordance  with  generally  accepted  government  auditing  standards. 
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Knowledge  at  Key  Junctures  Is  Critical  to  a 
Successful  Transition  to  Production 


Product  development  can  be  characterized  as  the  reduction  of  risk  and 
resolution  of  unknowns  through  the  acquisition  of  knowledge.  Compared 
with  the  DOD  programs  we  reviewed,  commercial  firms  gain  more 
knowledge  about  a  product’s  performance,  producibility,  and  ability  to 
meet  customer  requirements  much  earlier  in  the  product  development 
process.  The  best  firms  will  not  launch  a  new  product  development  unless 
they  have  high  confidence  that  they  have  achieved  a  match  between  what 
the  customer  wants  and  what  the  firms  can  deliver.  The  firms’  keen 
knowledge  of  their  technological  capabilities  and  limitations  frames  the 
design  features  they  are  willing  to  offer.  Before  the  midway  point  in 
product  development,  the  firms  attain  enough  knowledge  to  ensure  that 
the  product  works.  By  the  end  of  product  development,  but  before 
fabrication  of  production  items  begins,  commercial  firms  prove  that  their 
production  methods  will  yield  the  desired  volume  and  quality  of  the 
product  within  the  desired  unit  cost.  Throughout  product  development, 
the  firms  adhere  to  their  own  proven  standards  as  to  what  constitutes  an 
acceptable  level  of  knowledge.  By  the  time  actual  production  begins,  few 
risks  or  unknowns  remain. 

The  DOD  programs  we  reviewed  allowed  more  technology  development  to 
carry  over  into  product  development.  Consequently,  they  proceeded 
through  product  development  with  much  less  knowledge  about  required 
technologies,  design  capability,  and  producibility  than  the  commercial 
firms  we  visited.  Two  programs — ^the  C-17  and  the  F-22 — did  not  or  will 
not  attain  the  same  level  of  knowledge  of  the  design’s  ability  to  perform  or 
be  produced  until  late  in  development  or  early  production.  Attaining  the 
match  between  product  design  and  customer  requirements  will  not  be 
certain  until  testing  is  completed  late  in  development,  a  practice  that  is  not 
acceptable  in  commercial  product  developments.  Proof  that  production 
methods  can  yield  the  desired  volume  and  quality  within  the  desired  unit 
cost  will  occur  after  production  begins.  As  a  result,  the  dod  programs  tend 
to  have  more  unknowns  and  greater  risks  to  manage  when  production 
fabrication  begins,  dod’s  review  mechanisms  tended  to  overstate 
knowledge  or  understate  risks,  as  borne  out  by  problems  or  unknowns 
discovered  later  in  product  development.  'Two  programs  we 
reAuewed — ^jdam  and  the  aim-9X — ^have  attempted  to  make  a  closer  match 
up  front  between  demonstrated  technology  and  customer  requirements. 
They  are  thus  in  a  better  position  to  gain  critical  knowledge  sooner  in  the 
remainder  of  their  development  phases. 
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Commercial  Firms 
Develop  Knowledge 
Sooner  in  the  Product 
Development  Cycle 


The  successful  management  of  cost,  schedule,  and  performance  risk  in 
transitioning  a  new  product  from  development  to  production  is  related  to 
how  soon  full  knowledge  about  key  dimensions  of  the  product  is  attained. 
In  this  sense,  knowledge  is  not  the  same  as  information.  Knowledge,  in 
this  context,  means  that  program  managers  and  decisionmakers  have 
reached  virtual  certainty  about  an  aspect  of  the  product  being  developed, 
such  as  a  critical  manufacturing  process.  It  is  the  inverse  of  risk. 
Information  is  essential  to  attaining  knowledge,  but  not  all  information 
produces  the  same  level  of  certainty.  For  example,  a  study  that  shows  a 
new  manufacturing  process  should  work  does  not  produce  the  same 
degree  of  certainty  that  actual  use  of  the  process  on  another  product  does. 
For  the  purposes  of  comparing  commercial  and  dod  product  development 
cycles,  we  have  characterized  the  point  at  which  virtual  certainty  of  some 
aspect  of  a  product  is  achieved  as  a  “knowledge  point.” 


The  commercial  and  military  programs  we  reviewed  did  not  all  follow  the 
same  processes  or  steps  in  their  development  cycles.  However,  at  some 
point,  full  knowledge  was  or  will  be  achieved  about  a  completed  product, 
regardless  of  what  development  approach  was  taken.  Knowledge  on 
product  developments  can  be  broken  down  into  three  points  or  junctures: 
when  a  match  is  made  between  the  customer’s  requirements  and  the 
available  technology;  when  the  product’s  design  is  determined  to  be 
capable  of  meeting  performance  requirements;  and  when  the  product  is 
determined  to  be  producible  within  cost,  schedule,  and  quality  targets. 
Program  launch  or  start  is  the  point  at  which  organizations  define  a 
product’s  performance,  cost,  and  schedule  estimates  and  commit  to 
making  the  financial  investment  needed  to  complete  development  and 
bring  the  product  into  production.  Figure  2.1  illustrates  the  three 
knowledge  points  and  the  differences  between  the  commercial  and 
military  product  developments  in  terms  of  when  they  attain  knowledge. 
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Figure  2.1:  Comparison  of  Three  Key  Knowledge  Points  for  Commercial  and  Military  Product  Developments 
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Knowledge  Point  1 :  Knowledge  that  a  match  exists  between  technology  and  requirements. 
Knowledge  Point  2:  Knowledge  that  the  design  will  work  as  required. 

Knowledge  Point  3:  Knowledge  that  the  design  can  be  produced  within  cost,  schedule,  and 
quality  targets. 
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Because  military  programs  tend  to  start  product  development  with  more 
unknowns,  it  takes  them  additional  time,  sometimes  until  well  after 
production  begins,  to  actually  discover  and  capture  enough  solid 
information  to  attain  full  product  knowledge  and  thereby  virtually 
eliminate  risk.  The  programs  continue  to  resolve  risks  about  the 
performance  of  the  product  design  and  its  producibiUty  after  they  begin 
production.  Commercial  firms  attain  full  product  knowledge  before 
fabrication  of  production  imits  begins. 

Decisions  made  in  matching  requirements  with  technology  have  a  direct 
impact  on  what  is  required  to  reach  subsequent  knowledge  points.  Synergy 
exists  when  a  decision  early  in  product  development,  such  as  establishing 
product  requirements,  reduces  the  amount  of  unknowns  that  have  to  be 
resolved  before  full  knowledge  of  product  design  performance  and 
producibiUty  is  attained.  Conversely,  a  decision  to  reach  for  a 
technological  advance  can  increase  the  effort  required  to  attain  full  design 
performance  knowledge,  which,  in  turn,  can  delay  the  attainment  of 
producibiUty  knowledge. 


Knowledge  Point  1: 
Requirements  and 
Technological 
Capability  Are 
Matched 


To  minimize  the  amount  of  technology  development  that  occurs  during 
product  development,  the  commercial  companies  we  visited  employ 
discipUned  processes  to  match  requirements  with  technological  capability 
before  product  development  begins.  The  process  reflects  production 
reaUties  and  demands  proof  that  the  technology  will  work  and  can  be 
produced  at  an  acceptable  cost,  on  schedule,  and  with  high  quality.  The 
companies  bring  soUd  technological  knowledge  to  the  requirements 
process  in  the  form  of  current,  high-fidelity  information  from  predecessor 
programs,  people  with  first-hand  experience  on  those  programs,  or  new 
technologies  deemed  mature  as  a  result  of  having  “graduated”  firom  a 
discipUned  technology  development  and  screening  process.  In  addition, 
the  companies  commimicate  extensively  with  customers  to  match  their 
wants  and  needs  to  the  firm’s  available  technology  and  its  ability  to 
manufacture  an  appropriate  product.  They  do  not  stray  far  from  their 
technological  foundation.  In  the  commercial  world,  this  process  is 
sometimes  referred  to  as  QuaUty  Functional  Deployment,  a  technique  used 
to  convert  complex  or  unclear  customer  requirements  into  design  and 
manufacturing  requirements. 

For  the  programs  we  reviewed,  dod  did  not  get  as  good  a  match  between 
customer  requirements  and  the  technology  that  could  be  confidently 
delivered.  UsuaUy,  the  individual  services  identify  a  need  for  a  capability 
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Commercial  Practices 


and  translate  that  need  into  specific  weapon  performance  objectives  and 
thresholds.  Even  though  dod  examines  the  potential  for  available 
technology  to  meet  product  requirements,  it  allows  requirements  to  drive 
technology  and  make  it  reach  beyond  what  is  proven,  dod  has  less 
knowledge  than  commercial  firms — ^in  the  form  of  predecessor  programs, 
people  with  experience  on  those  programs,  and  technology 
maturity— about  the  ability  of  the  design  to  deliver  required  performance 
and  be  produced  at  the  point  it  starts  a  program.  The  knowledge  that  the 
match  has  been  achieved  between  required  performance  and  the 
capability  of  technology  is  often  not  attained  until  after  production  begins, 
when  operational  testing  is  completed. 


We  found  examples  of  best  commercial  practices  for  matching 
requirements  to  technology  at  Boeing,  Hughes,  and  Ford.  Boeing 
communicated  with  the  airlines  to  set  achievable  requirements  for  the 
777-200  airplane  and  tested  the  design  early.  Hughes  used  a  technology 
development  process  that  graduated  new  technologies  from  concept  into  a 
product  development  program,  enabling  the  firm  to  make  what  it  saw  as 
quantum  performance  increases  with  mature  technologies.  Ford  uses  its 
technology  deployment  process  to  separate  immature  technology  from  a 
new  product’s  development.  It  enforces  a  decision  point  in  product 
development  in  which  new  technology  must  show  proof  that  it  will  meet 
cost  and  performance  expectations  to  be  accepted  onto  a  product’s 
development  program. 

Boeing  established  requirements  for  the  777-200  airplane  after  extensive 
communications  with  several  airlines  over  an  18-month  period  before 
beginning  product  development.  Originally,  the  airlines  thought  that  a  767 
derivative  would  serve  their  purposes.  However,  during  the  course  of 
discussions,  Boeing  and  the  airlines  realized  that  a  “family”  of  airplanes 
that  would  provide  the  airlines’  range  and  payload  flexibility  was  required. 
During  that  time,  Boeing  matched  customer’s  needs  to  its  technological 
capability  using  an  airline  working  group  made  up  of  airline 
representatives  to  estabhsh  market  and  product  requirements  and 
extensive  performance  testing  to  these  requirements.  Boeing  tested  all  of 
the  airplane’s  various  systems  together  as  a  single,  integrated  entity  in 
simulated  flight  conditions  very  early  in  product  development.  In  addition, 
Boeing  completed  an  initial  design  review  that  ensured  the  design’s 
performance,  identified  all  risks,  made  plans  for  their  resolution,  and 
selected  a  final  configuration. 
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The  diligence  Boeing  applied  to  matching  requirements  and  technological 
capability  on  the  777  was  a  practice  that  resulted  from  painful  experiences 
on  two  predecessor  programs.  On  one  program,  Boeing  had  erred  on  the 
side  of  proposing  too  many  new  technologies  and  design  features  on  a 
new  aircraft  design.  Boeing  halted  the  program  before  the  launch  point, 
after  one  airline  customer  complained  that  it  contained  “too  much 
technology  for  technology’s  sake.”  In  retrospect,  Boeing  was  relieved 
because  the  design  contained  too  much  risk  to  make  the  sound  business 
case  needed  for  the  launch  decision.  On  another  program,  Boeing  had 
erred  too  much  on  the  side  of  allowing  customers  to  add  requirements 
after  program  launch.  This  program  was  completed,  and  aircraft  were 
delivered;  however,  delays  occurred  in  the  development  phase  due  to 
reworking  the  design,  and  the  early  production  aircraft  e3q)erienced 
problems  with  reliability  rates.  The  requirements  process  for  the  777-200 
applied  the  lessons  learned  from  both  programs.  Figure  2.2  shows  the 
Boeing  777  airliner. 
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Lessons  learned  from  previous  programs  helped  Boeing  match  technology  with  requirements 
before  launching  the  111. 

Source:  Boeing  Commercial  Airplane  Group. 


Because  of  early  communication  and  testing,  Boeing  met  the  airlines’ 
requirement  with  plans  for  a  family  of  airplanes  in  the  medium-lift 
category  that  would  meet  the  airlines’  needs  in  a  timely  manner.  The 
777-200  model  was  first,  followed  by  models  that  would  increase  capacity 
and  range.  Boeing  also  found  that  it  could  meet  requirements  for  the 
777-200  using  mostly  existing  technologies.  When  new  technologies  were 
needed,  such  as  digital  avionics  and  advanced  materials,  they  had  to  be 
demonstrated  with  laboratory  testing,  simulation,  and  modeling  before 
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introducing  them  to  the  development  program.  Boeing  also  decided 
against  some  new  technology  on  the  777  as  the  result  of  its  matching  of 
requirements  to  capability.  For  example,  Boeing  originally  designed  the 
airplane  using  anew,  lightweight  material — aluminum  lithium — ^for  some 
portions.  However,  at  the  design  review  that  preceded  the  start  of 
development,  manufacturing  members  on  the  design/build  team  argued 
that  the  material  was  expensive  and  hard  to  find  and  that  the  fabrication 
processes  for  it  were  not  yet  fully  understood.  Despite  the  increase  in 
weight  that  would  result,  the  team  decided  against  using  the  material. 
Boeing  made  the  customers  aware  of  the  problem  and  worked  with  them 
to  reduce  the  weight.  Ultimately,  the  final  777-200  configuration  was 
1,500  pounds  under  its  originally  proposed  weight 

According  to  Hughes  Space  and  Communications,  its  newest  product,  the 
HS-702  satellite,  was  a  quantum  leap  in  satellite  technology.  It  provided 
twice  as  much  power  and  capacity  as  its  predecessor;  however,  it  was 
made  with  mature  technology.  Hughes  used  its  technology  development 
process  to  demonstrate  new  technology  before  inserting  it  into  the 
HS-702’s  development.  Specifically,  it  uses  Technology  Roadmaps  to 
prioritize  technology  against  business  needs  and  corporate  strategies,  and 
investments  are  made  accordingly.  Product  development  managers  will 
not  use  the  resultant  technology  imtil  it  proves  capable  of  meeting  product 
performance,  cost,  schedule,  and  quality  requirements  and  manufacturing 
processes  are  in  place  and  in  control. 

Hughes  identified  three  technologies  it  considered  high  risk  when  it  began 
development  of  the  HS-702  satellite:  the  gallium  arsenide  solar  cells  and 
solar  concentrators,  the  xenon-ion  propulsion  system,  and  the  deployable 
radiator  needed  to  radiate  additional  heat  resulting  from  the  increased 
power.  The  new  solar  cell  and  propulsion  system  technologies  had  been 
through  the  corporate  technology  development  process  and  had  been 
applied  on  predecessor  programs,  so  there  was  significant  knowledge 
about  them  at  the  outset  The  deployable  radiator  was  high  risk  because 
Hughes  did  not  believe  it  had  the  technology  for  this  capability.  Hu^es 
foimd  a  Russian  supplier  that  was  already  producing  the  required 
technology,  greatly  reducing  technical  risk.  At  Hughes,  technology  that  is 
not  proven  through  predecessor  programs  or  the  development  process  is 
not  allowed  onto  a  new  product  For  example,  Hughes  wanted  to  include 
lithium  ion  batteries  and  advanced  receiver  technologies  on  the  HS-702 
satellite  to  decrease  weight  and  improve  its  capacity  and  power,  but  it  did 
not  include  them  because  these  technologies  had  not  met  cost,  schedule, 
performance,  and  quality  targets. 
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Ford  reduces  risk  from  new  technology  during  product  development  with 
its  Technology  Deployment  Process.  This  process  prioritizes,  matures,  and 
introduces  new  or  emerging  technologies  for  product  development, 
reducing  risk  from  the  time  the  concept  is  proposed  to  its  ultimate 
implementation  on  a  new  product.  During  this  process,  technologists 
agree  to  deliver  new  technology  to  vehicle  centers  for  product 
development.  Managers  of  product  development  are  customers  to 
technologists  and  can  agree  or  refuse  to  sponsor  technology,  which 
creates  a  shared  understanding  of  deliverables  between  the  technologist 
and  the  product  manager.  Each  technology  project  has  a  concept 
readiness  milestone,  which  includes  a  demonstration  of  the  hardware  and 
its  required  manufacturing  processes,  as  well  as  testing  and  analysis 
results.  Also,  each  technology  has  an  implementation  readiness  milestone, 
which  includes  proof  that  the  technology  is  ready  for  product 
development  with  identified  and  manageable  risks.  At  this  point,  the  new 
technology’s  cost,  schedule,  and  quality  targets  must  be  met. 

Ford  enforces  a  “Wall  of  Invention”  at  the  start  of  each  new  product’s 
development.  Beyond  this  point,  no  new  technologies  or  design  features 
are  allowed  to  be  added  to  the  vehicle.  From  then  on,  the  product  manager 
takes  responsibility  for  successfully  developing  and  producing  the  new 
vehicle.  Product  managers  are  veterans  of  previous  production  programs, 
and  their  production  knowledge  makes  them  very  discriminating  in 
deciding  what  new  technology  will  be  allowed  onto  the  vehicle. 


DOD  Practices  dod  programs  did  not  attain  a  match  between  technology  and 

requirements  at  the  time  of  launch,  dod  accepted  varying — but 
consistently  higher — degrees  of  technological  risk  on  four  of  the  product 
development  programs  we  reviewed.  Technologies  ranged  from 
comparatively  high-risk  advances  for  the  F-22’s  avionics  and  low 
observability  to  low-risk  advances  for  mostly  off-the-shelf  components 
being  used  to  develop  the  aim-9X  missile  and  jdam.  The  C-17  program  was 
developed  using  mostly  existing  technology,  with  some  demanding 
features  required  of  the  airframe  design,  dod  practices  for  matching 
product  requirements  to  available  technology  on  the  aim-9X  and  jdam 
appeared  to  do  a  better  job  of  reducing  risks. 

The  F-22  development  program  has  reduced  risk  from  new  technologies 
across  most  of  its  key  design  features  through  extensive,  high-fidelity 
modeling  and  analysis.  However,  the  program’s  product  development 
includes  new  low-observable  materials,  avionics,  and  propulsion 
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technology  whose  performance  is  undemonstrated  and  raises  some 
concern  about  producibility.  Low  observability  is  one  of  the  F-22’s  most 
important  characteristics  and  is  dependent  on  many  factors,  such  as  the 
shape  of  the  airframe,  its  flight  characteristics,  and  airframe  materials  and 
coatings.  Although  the  program  has  done  pole  model  testing  of  the  aircraft 
to  determine  whether  it  meets  its  low-observability  requirements,  the 
aircraft  includes  10  newly  developed  derivatives  of  existing  materials  that 
are  important  to  its  low-observability  feature.  The  materials  will  have  had 
3  years  of  flight  testing  before  they  are  applied  to  the  first  production 
aircraft;  however,  the  performance  and  maintainability  of  these  materials 
cannot  be  completely  verified  until  radar  signatures  can  be  tested  in 
varying  climates.  This  testing  is  scheduled  to  be  completed  2  years  after 
production  begins. 

Similarly,  much  remains  to  be  proven  on  the  F-22’s  avionics  technology 
before  it  is  certain  that  the  technology  is  a  match  for  the  performance 
requirements.  The  avionics  software  features  a  level  of  integration  not 
previously  achieved  in  a  fighter.  One  msgor  technical  challenge  is 
perfecting  the  way  in  which  various  avionics  subsystems  will  interface. 
The  program  has  incorporated  a  series  of  risk  reduction  activities 
designed  to  mitigate  these  software  integration  problems.  For  example,  it 
has  defined  and  replicated  all  of  the  avionics  interfaces  at  this  point  in 
development,  but  this  testing  will  not  prove  that  the  various  subsystems 
will  interact  in  identifying  and  prioritizing  threats.  The  program  has  begun 
testing  in  an  avionics  integration  laboratory  and  on  a  flying  avionics  test 
bed  mounted  on  a  Boeing  757  and  will  further  test  the  avionics  capabilities 
for  prioritizing  threats  at  very  high  densities  and  conditions  once  the  test 
facility  is  ready.  However,  until  that  type  of  testing  and  flight  testing  begin 
to  produce  results — ^in  1999  at  the  earliest — ^there  are  significant 
unknowns  concerning  the  capability  of  integrated  avionics  on  the  F-22. 
The  program  plans  three  incremental  updates  to  the  software,  completing 
its  development  in  2002.  By  this  time,  Lockheed  plans  to  have  at  least 
40  production  F-22s,  an  investment  of  at  least  $8  billion,  finished  or  in  the 
manufacturing  process. 

The  F-22  engines  have  many  advanced  features  to  meet  aggressive 
performance  requirements.  One  such  requirement  is  for  supercruise — ^the 
ability  to  sustain  supersonic  speeds  without  the  use  of  afterburners.  Some 
of  these  advanced  features  resulted  in  components  that  are  more  difficult 
to  produce  than  originally  anticipated  and  are  resulting  in 
higher-than-planned  recurring  production  costs.  The  producibility  risk  is 
largely  driven  by  unknowns  concerning  the  hoUow  fan  blades,  some 
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components  fabricated  from  composite  materials,  the  turbine  exhaust 
case,  and  a  few  features  of  the  exhaust  nozzle.  A  product  cost  reduction 
process  has  been  implemented  to  mitigate  these  risks,  but  program 
officials  acknowledged  that  the  user’s  overall  performance  requirements 
for  the  engines  remain  very  challenging,  and  it  is  still  unclear  whether  the 
engines  will  meet  all  of  the  requirements. 

For  the  most  part,  the  C-17  was  developed  using  nondevelopmental  items 
or  commercial  parts.  However,  performance  requirements  for  short 
landing  and  takeoff,  payload,  and  range  required  design  innovations  that 
significantly  increased  technical  risk  on  the  program.  A  new  technology 
that  added  risk  to  the  program  was  aluminum  lithium,  a  new,  unproven 
alloy  that  the  program  used  for  its  strength  and  weight  savings.  The  first 
production  aircraft  contained  2,200  pounds  of  the  alloy.  Its  application 
was  unsuccessful  due  to  alack  of  knowledge  about  the  technology’s 
characteristics.  The  program  discontinued  its  use  because  of  its  potential 
to  warp  during  handling,  cracking  and  chipping  during  installation,  the 
need  to  keep  it  separate  from  other  materials,  and  concerns  over  sources 
of  supply.  It  is  being  phased  out  of  the  program  and  the  51st  production 
airplane  will  be  fi:ee  of  it.  The  C-17  program  replaced  aluminum  lithium 
with  aluminum  alloy  7050,  a  less  expensive  and  easier-to-handle  material 
with  several  sources  of  supply.  Boeing  made  the  same  decision  at  the 
outset  of  its  777-200  development  program  based  on  similar  conclusions 
reached  by  its  design/build  team.  This  is  a  good  illustration  of  different 
knowledge  standards  applied  in  making  design  decisions.  Figure  2.3  shows 
the  C-17  aircraft 
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Figure  2.3:  C-17  Cargo  Aircraft 


Although  the  C-17  was  developed  mostly  with  nondevelopmental  items  and  commercial  parts, 
performance  requirements  significantly  increased  technical  risk. 

Source:  DOD. 


Also,  the  software  development  effort  to  meet  avionics  performance 
requirements  turned  out  to  be  significantly  more  complex  than  the  Air 
Force  thought.  When  the  C-17  development  program  began  in  1985,  for 
example,  the  Air  Force  had  identified  4  subsystems  with  about  164,000 
lines  of  code  that  had  to  be  developed.  By  1990,  this  number  had  increased 
to  56  subsystems  and  about  1,356,000  lines  of  code,  including 
approximately  643,000  newly  developed  lines  of  code. 
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The  JDAM  and  aim-9X  development  programs  have  been  more  attentive  to 
matching  customer  requirements  with  technological  capabilities.  Both 
programs  chose  designs  that  relied  strongly  on  predecessor  data,  and  both 
programs  emphasized  affordability  as  a  key  product  requirement,  jdam 
used  modified  variants  of  proven  product  lines  for  its  guidance  component 
and  global  positioning  system.  It  also  used  mature,  existing  components 
from  other  proven  manufacturing  processes  for  its  own  system  for 
controlling  tail  fin  movements.  The  designs  for  the  battery  and  the  tail 
housing  both  use  mature  technology  and  will  be  built  using  mostly  existing 
tooling  and  processes.  The  program  office  and  contractor  have  been  able 
to  further  minimize  risk  by  conducting  extensive  development, 
qualification,  and  flight  testing  to  demonstrate  performance,  quality,  and 
reliability  on  some  key  subsystems. 

Except  for  its  tracking  technology,  the  aim-9X  air-to-air  missile  will  use 
technology  largely  from  manufacturing  processes  already  in  place  or  from 
the  United  Kingdom’s  Advanced  Short-Range  Air-to-Air  Missile  production 
program.  For  example,  the  sensor  is  currently  being  developed  for  the 
United  Kingdom  missile,  and  over  2,500  sensors  will  have  been  produced 
before  the  first  aim-9X  is  assembled.  The  aim-9X  airfirame  is  adapted  from  a 
past  Air  Force  flight  test  program  and  is  made  up  of  proven  technology. 
The  rocket  motor,  engine,  warhead,  and  fuse  are  all  taken  directly  from 
AIM-9M  missiles  and  are  already  in  production.  The  aim-9X  tracking  system 
is  the  primary  area  of  new  technology.  It  is  a  sophisticated,  computerized 
component  that  translates  the  light  image  obtained  by  the  seeker  into 
electronic  signals  that  can  be  viewed  by  the  pilot,  discriminate  against 
countermeasures,  and  guide  the  missile  to  a  particular  point  on  the  target 
aircraft.  It  provides  the  seeker  a  wider,  higher  fidelity  view  in  which  to  find 
targets.  This  new  technology  has  been  in  development  for  some  time  and 
is  made  up  of  off-the-shelf  components.  Hughes  believes  this  risk  is 
manageable  because  of  modeling  and  simulation  that  preceded  the 
development  program. 


Knowledge  Point  2: 
The  Design  Will 
Perform  as  Required 


The  completion  of  engineering  drawings  and  their  release  to 
manufacturing  organizations  signify  that  program  managers  are  confident 
in  their  knowledge  that  the  design  performs  acceptably  and  can  be 
considered  mature.  The  drawings  are  critical  to  documenting  this 
knowledge  because  they  are  not  only  precision  schematics  of  the  entire 
product  and  all  of  its  component  parts — they  also  reflect  the  results  of 
testing  and  describe  the  materials  and  manufacturing  processes  to  be  used 
to  make  each  component.  Both  dod  and  commercial  companies  consider 
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the  design  to  be  essentially  complete  when  about  90  percent  of  the 
engineering  drawings  are  completed.  Both  sectors  schedule  a  critical 
design  review  (cdr)  to  review  the  drawings,  confirm  the  design  is  mature, 
and  “freeze”  it  to  minimize  changes  in  the  future.  Figure  2.4  compares 
what  knowledge,  in  the  form  of  released  drawings,  was  in  hand  at  the  time 
of  CDR  for  the  commercial  and  dod  programs  we  reviewed. 


Coinmercial  Practices  The  commercial  firms  we  visited  released  over  90  percent  of  the  products’ 

engineering  drawings  at  cdr.  At  that  point — about  midway  through 
development— the  firms  had  near  certainty  that  their  product  designs 
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would  meet  requirements  and  had  gone  a  long  way  to  ensure  that  the 
product  could  be  produced. 

Boeing  had  full  knowledge  of  the  777-200  design’s  capability  at  its  final 
CDR.  It  began  releasing  engineering  drawings  after  a  preliminary  design 
review  when  product  development  began  in  1990  and  completed  this 
release  process  in  1992,  less  than  2  years  later,  cdr  at  Boeing  is  generally 
done  by  peer  organizations,  such  as  engineers  firom  another  program. 
Outside  consultants  are  sometimes  used.  Boeing  officials  told  us  that,  once 
the  design  drawings  were  released  to  the  manufacturing  process,  they 
averaged  less  than  two  changes  per  drawing,  well  below  their  average  on 
past  programs  and  exceeding  their  goal  for  the  777-200  program.  Once  cdr 
was  complete,  Boeing  strictly  enforced  the  design  freeze  for  the  777-200. 
For  example,  Boeing  incorporated  a  customer  requirement  to  include 
folding  wingtips,  along  with  the  supporting  bulkheads,  into  the  777  design 
at  a  cost  of  nearly  $40  million.  Later,  the  customer  decided  the  folding 
wingtips  were  not  necessary;  however,  Boeing  left  the  bulkheads  in  the 
wings  anyway  because  all  of  the  engineering  drawings  were  completed 
and  the  risk  of  introducing  changes,  even  though  the  changes  would  have 
saved  weight,  was  considered  too  high  relative  to  cost  and  schedule 
targets. 

Hughes,  which  designs  both  military  and  commercial  satellites  in  the  same 
facility,  completed  two  design  reviews  for  development  of  the  HS-702. 
First,  it  passed  an  internal  review  to  satisfy  the  coiporation  that  its  basic 
design  was  producible.  Second,  it  reiterated  design  reviews  on  individual, 
customized  satelUtes  as  they  were  ordered  by  customers.  During  the 
internal  development  process,  Hughes  began  releasing  engineering 
drawings  at  preliminary  design  review  and  completed  releasing  drawings 
at  CDR,  about  15  months  into  a  26-month  product  development  process. 
Hughes’  goal  was  to  complete  and  release  all  engineering  drawings  at  cdr; 
however,  representatives  told  us  the  drawings  were  close  to  95  percent 
released  at  that  point. 

Chrysler  described  several  practices  for  ensuring  mature  designs.  It  uses 
information  from  predecessor  products  to  quickly  develop  focused 
updates  and  innovative  derivatives  to  existing  products.  This  predecessor 
data,  along  with  its  use  of  platform  teams,  allows  Chrysler  to  take 
advantage  of  existing  information  and  expertise  in  new  product 
developments.  It  is  able  to  simultaneously  develop  and  test  the  product’s 
design  and  its  required  processes,  which  it  described  as  true  concurrent 
engineering  at  the  right  time.  Chrysler  uses  prototypes  to  refine  the 
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product  and  capture  design  knowledge  early.  It  also  employs  a  pilot 
production  process  and  extensive  testing  facilities  at  its  technical  center  to 
prove  the  design’s  performance  and  producibility  before  it  is  released  to 
manufacturing  facilities. 


The  C-17  and  the  F-22  programs  had  less  knowledge — ^in  the  form  of  test 
results  or  engineering  drawings — about  their  designs  than  commercial 
companies  did  at  the  time  they  held  their  cdrs.  The  programs  did  not  get 
or  were  not  projected  to  get  to  the  same  level  of  completion  on  the 
drawings  until  later  in  the  development  cycle,  which  placed  greater 
reliance  on  the  lesser  information  available  at  cdr.  This  is  important 
because  the  review  is  a  major  event  that  represents  a  point  of  departure 
from  detail  design  to  manufacture  of  the  product.  The  risks  of  proceeding 
with  CDR  and  the  rest  of  development  as  planned  are  increased  without  the 
requisite  drawings  in  hand.  This  was  deemed  acceptable  for  both 
programs. 

The  C-17  program  had  released  10,229  of  its  engineering 
drawings — 56  percent— at  cdr  in  August  1988,  just  past  the  midway  point 
in  development.  After  cdr,  an  additional  8,161  drawings  were  released, 
over  4,000  of  which  were  released  after  assembly  of  the  first  production 
aircraft  had  started.  We  estimate  that  the  C-17  program  did  not  release 
95  percent  of  its  initial  drawings  until  December  1991,  more  than  3  years 
after  cdr  and  after  seven  production  aircraft  had  been  delivered  to  dod. 
The  C-17  encountered  numerous  technical  problems  during  testing  that 
resulted  in  redesigns,  cost  increases,  and  schedule  delays.  For  example, 
flight  tests  in  1991  revealed  that  an  innovative  system  of  blown  wing 
flaps — essential  to  the  C-17’s  ability  to  fly  steep  landing  approaches  at 
slow  speeds  to  access  shorter  runways — ^suffered  heat  damage  and 
acoustical  cracks.  To  correct  this  problem,  the  contractor  had  to 
completely  redesign  the  slats  to  include  titanium  skin  and  substructure. 
This  knowledge  was  not  obtained  until  the  program  was  at  least  two-thirds 
into  development.  Also,  in  1992  both  wings  on  the  static  test  article  failed 
at  approximately  128  percent  during  a  150-percent  limit  load  test  and  had 
to  be  redesigned. 

Software  development  and  management  problems  also  contributed  to  C-17 
cost  and  schedule  problems.  The  Air  Force  lacked  specific  knowledge 
about  software  development  problems  as  they  occurred.  The  first  C-17 
aircraft  achieved  first  flight  19  months  later  than  planned  and  did  not 
include  many  of  the  mission-critical  software  functions  required  for  a  fully 


Page  36 


GAO/NSIAD“98'56  Best  Practices 


Chapter  2 

Knowledge  at  Key  Junctures  Is  Critical  to  a 
Successful  Transition  to  Production 


operational  aircraft.  Problems  such  as  these  caused  completion  of  testing 
on  the  C-17  to  be  delayed  by  well  over  1  year. 

The  F-22  program  had  released  3,070  initial  structures  and  systems 
drawings,  less  than  one-third  of  the  eventual  drawings,  at  its  cdr  held  at 
the  midway  point  in  development.  Since  then,  an  additional  6,032  drawings 
have  been  completed  and  released.  The  drawing  release  process  is  not  yet 
complete  for  the  F-22,  so  we  cannot  determine  if  the  program  has  yet 
released  95  percent  of  its  drawings.  F-22  production  is  scheduled  to  begin 
in  June  1999. 

Even  though  it  is  stUl  too  early  to  predict  outcomes  on  the  aim-9X  missile 
and  the  jdam  development  programs,  their  prospects  appear  promising 
because  they  have  chosen  mostly  proven  technology  from  existing 
programs  to  achieve  performance  requirements.  They  appear  to  have 
robust,  mature  designs  that  will  use  manufacturing  processes  already 
identified  and  demonstrated  to  be  stable  or  in  control,  jdam  held  its  cdr  in 
August  1995.  At  that  time,  it  had  released  65  percent  of  its  drawings.  In 
addition,  it  demonstrated  how  the  product  wordd  meet  requirements  using 
data,  analysis,  and  a  physical  display  of  the  product.  The  time  it  took  to 
build  up  and  load  a  jdam  was  also  demonstrated.  aim-9X  has  not  yet  held  its 
CDR,  but  program  officials  indicated  that  they  will  have  already  built  a 
prototype  before  cdr  and  hope  to  freeze  the  design  based  on  that 
prototype. 


Knowledge  Point  3: 
Production  Units  Will 
Meet  Cost,  Quality, 
and  Schedule 
Objectives 


The  companies  we  visited  reached  the  point  at  which  they  knew  that 
manufacturing  processes  would  produce  a  new  product  according  to  cost, 
quality,  and  schedule  targets  before  they  began  fabricating  production 
articles.  This  meant  more  than  knowing  the  product  could  be 
manufactured;  it  meant  that  all  key  processes  were  under  control,  so  the 
quality,  volume,  and  cost  of  their  output  were  proven  and  acceptable.  The 
C-17,  in  production  for  7  years,  stUl  does  not  have  all  of  its  processes 
imder  control.  The  F-22  program  is  not  scheduled  to  have  all  of  its 
processes  under  control  until  the  4th  year  of  production.  The  aim-9X  and 
JDAM  programs  appear  to  be  doing  better  in  this  regard.  Both  programs  are 
using  the  same  approach  as  commercial  firms  to  attempt  to  get  their 
critical  manufacturing  processes  in  control  before  the  programs  enter 
production.  Figure  2.5  illustrates  the  difference  between  the  commercial 
HS-702  and  777-200  programs  and  the  military  C-17  and  F-22  programs 
concerning  when  full  knowledge  about  the  production  processes  was 
achieved. 
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Figure  2.5:  Comparison  of  When  Commercial  and  DOD  Programs  Achieve  Knowledge  That  Processes  Can  Produce  an 
Acceptable  Product  _ _ _ 


Hughes  HS-702 


Being  777-200 


Program  Launch 


Knowledge  Point  3 


Production  Start 


DOD  F-22 


DOD  C-17 


Program  Launch 


Production  Start 


Knowledge  Point  3  will 
be  reached  well  after 
production  begins. 

Knowledge  Point  3  has 
not  yet  been  reached. 


Unknowns 


Knowns 


Commercial  Practices  The  commercial  firms  relied  on  existing  manufacturing  processes  and 

statistical  process  control  (spc)  data  to  achieve  this  knowledge  in  a  timely 
manner  and,  in  fact,  had  all  their  key  processes  under  statistical  control 
when  production  began,  spc  is  established  by  monitoring  processes  to  see 
if  they  are  consistently  producing  output  that  is  within  the  quality 
standards  and  tolerances  set  for  the  overall  product.  Statistics  concerning 
the  quality  of  each  process  output  are  analyzed,  and  when  the  output  is 
out  of  tolerance,  process  owners  search  for  causes.  Once  a  process  is 
producing  consistently  high-quality  output,  the  process  is  considered  to  be 
in  statistical  control,  and  inspections  can  be  reduced.  The  knowledge 
gained  using  spc  is  significant  in  transitioning  from  development  to 
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production  because  it  helps  companies  ensure  that  cost,  schedule,  quality, 
and  reliability  targets  will  satisfy  the  customer  and  the  business  case. 

The  ability  to  establish  spc  for  key  processes  before  production  began  was 
the  culmination  of  all  the  practices  employed  to  identify  and  reduce  risk. 
The  criteria  commercial  companies  had  established  throughout  their 
product  development  processes  forced  program  managers  to  prove  that 
the  product’s  design  was  capable  and  producible  early  in  the  process.  For 
example,  Boeing  carried  out  more  than  25  technical  and  manufacturing 
reviews  during  the  development  process  leading  up  to  production  to 
ensure  that  the  product’s  design  was  producible.  Boeing  identified 
computer-aided  three-dimensional  interactive  application  software  (catia) 
and  its  design/build  teams  as  crucial  to  ensuring  that  processes  were  in 
control  and  the  transition  to  production  went  smoothly.^  catia  software 
allowed  workers  to  share  design  and  process  information  about  the  777  in 
real  time  and  different  locations.  Manufacturing  engineers  could  check  on 
a  part’s  producibility  at  any  time  in  the  development  process. 

Hughes  told  us  that  it  knew  that  all  of  the  processes  to  manufacture  the 
HS-702  were  in  control  because  the  processes  were  either  already  in  place 
firom  previous  programs  and  under  spc  or  they  had  been  proven  to  be  in 
control  before  being  released  to  product  development  by  demonstrations 
during  the  corporation’s  overall  technology  development  process. 

Rgure  2.6  shows  the  Hughes  HS-702  satellite.  Ford  had  similar  practices  to 
Hughes,  often  demonstrating  spc  on  technology  before  it  was  inserted  into 
a  product  development,  even  before  production  begins. 


^CATIA  allows  designers  and  manufacturers  the  opportunity  to  view  design  drawings  as  three 
dimensional  and  provides  real-time  information  to  everyone  in  the  process  about  the  impact  a  change 
to  a  drawing  will  have  on  the  overall  design  and  manufacture  of  the  product. 
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Figure  2.6:  Hughes  HS-702  Satellite 


Hughes  has  achieved  statistical  process  control  on  the  low-volume  HS-702  satellite  before 
beginning  production. 

Source:  Hughes  Space  and  Communications. 


Chiysler’s  use  of  platform  teams,  catia  software,  and  its  technical  center 
all  contribute  to  a  smooth  transition  to  production  and  ensure  that 
processes  are  in  control  for  eveiy  new  product.  Perhaps  as  important, 
synergy  develops  with  a  disciplined  product  development  process  that 
accumulates  so  much  knowledge  up  to  this  point.  All  of  the  companies  we 
visited  agreed  that  knowledge  about  technology  and  design  up  front  in  the 
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process  makes  the  control  of  processes  possible  and  the  transition  to 
production  smooth. 


DOD  Practices  The  dod  programs  we  reviewed  demanded  less  proof  than  commercial 

firms  of  a  design’s  producibility  before  a  product  transitions  to  production. 
The  C-17  did  not  achieve  spc  on  all  key  processes  before  production  and 
will  not  reach  that  point  until  well  into  production.  The  F-22  is  not  slated 
to  achieve  control  on  all  key  processes  until  after  production  begins. 
Moreover,  basic  producibility  problems  were  not  discovered  until  late  in 
the  F-22’s  development  and  after  production  began  on  the  C-17.  These 
risks  went  unrecognized  even  though  both  programs  had  established 
criteria  for  ensuring  that  risks  were  acceptable  and  enough  knowledge  had 
been  gained  before  proceeding  to  the  next  program  phase.  These  exit 
criteria  were  light  on  production-related  requirements,  even  at  the 
decision  point  for  proceeding  with  production.  For  example,  the  F-22 
program’s  exit  criteria  for  moving  to  full  production  does  not  require 
information  from  the  final  production  readiness  review  until  after  the 
decision,  and  the  C-17  program’s  exit  criteria  did  not  require  proof  of 
producibility.  Neither  set  of  criteria  contained  requirements  about  drawing 
releases  nor  the  schedule  for  achieving  control  on  key  processes. 

The  C-17  program  began  production  in  1989  with  less  than  13  percent  of  its 
key  manufacturing  processes  in  control,  despite  completing  production 
readiness  reviews  that  were  intended  to  reduce  producibility  problems. 
The  program  has  produced  and  delivered  nearly  40  aircraft  and  has  now 
identified  420  key  manufacturing  processes.  Of  those,  only  56  are  under 
statistical  control.  This  inability  to  control  manufacturing  processes  was 
caused,  in  part,  by  an  immature  design.  For  example,  the  C-17  program 
discovered  major  design  changes  of  the  wings,  flaps,  and  slats  were 
required  after  cde,  as  prototype  aircraft  were  being  built.  In  addition  to 
causing  high  rates  of  drawing  changes,  these  late  design  changes  caused 
problems  on  the  production  line.  They  necessitated  costly  changes  to 
processes;  forced  the  manufacturers  to  develop  workaround  plans  that  led 
to  labor  inefficiencies;  and  resulted  in  high  rates  of  scrap,  rework,  and 
repair.  The  C-17  program  estimates  scrap,  rework,  and  repair  costs  on  the 
34  delivered  aircraft  to  be  over  $200  million. 

The  F-22  program  is  ahead  of  the  C-17  at  this  point  regarding  spc,  perhaps 
due  in  part  to  its  use  of  catia.  Of  926  key  manufacturing  processes 
identified,  almost  40  percent  of  its  key  processes  are  in  control  now, 

2  years  before  production  is  scheduled  to  begin.  Nonetheless,  the 
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production  preparations  for  the  F-22  illustrate  the  limitation  of  a  review 
mechanism  when  a  substantial  amount  of  knowledge  is  unattained.  The 
program’s  initial  production  readiness  review,  held  in  1995  when  only 
about  one-third  of  the  engineering  drawings  were  released,  did  not  report 
any  high  risks  in  manufacturing  or  producibUity,  although  clearly  much 
was  unknown  at  the  time  and  drawing  release  was  slower  than  called  for 
by  DOD  guidance.  In  1996,  an  independent  team  mandated  by  the  Air  Force 
reviewed  the  program  and  discovered  numerous  manufacturing  and 
producibility  problems,  such  as  underestimated  complexity  in 
manufacturing  processes,  understated  labor  requirements  for  building  the 
aircraft,  immature  definition  of  avionics  flight  test  requirements,  and 
concerns  about  software  integration.  These  and  other  concerns  led  to 
increases  to  development  and  production  cost  estimates  of  $2.1  billion  and 
$13  billion,  respectively.  Air  Force  officials  believe  that  these  additional 
costs  can  be  offset  by  reprogramming  existing  funds  and  by  implementing 
a  number  of  cost  avoidance  measures  in  the  future. 

The  JDAM  program  office  completed  manufacturing  assessments  at  the 
prime  contractor  and  all  major  suppliers  in  February  1997  in  such  areas  as 
process  controls  and  plant  capacities.  These  assessments  found  that  all  of 
the  production  tooling  was  in  place  to  build  jdam.  The  program  office  has 
identified  84  manufacturing  processes  that  must  be  in  control  to  meet  the 
product’s  key  performance  characteristics  and  has  achieved  statistical 
control  on  69  percent  of  the  processes  at  this  point  in  the  program,  about 
1  year  before  full-rate  production  begins.  The  aim-9X  program  is  still  very 
early  in  its  emd  phase  but  has  already  identified  aU  of  the  critical 
manufacturing  processes  that  must  be  in  control  to  meet  the  product’s  key 
performance  characteristics.  The  contractor  told  us  that  all  of  those 
processes  are  stable  at  this  point  in  the  program,  and  most  are  in  statistical 
control. 

Although  the  F-22  program  may  not  be  characterizing  and  controlling  its 
manufacturing  processes  as  quickly  as  commercial  firms  do,  it  offers 
perspective  on  some  of  the  strides  that  dod  has  made  in  preparing 
weapons  for  production  since  the  1980s.  In  1985,  we  reported  that  several 
weapons  had  encountered  substantial  problems  in  early  production 
because  they  used  hand-built  development  prototypes  made  in  special 
shops  by  expert  personnel^  Production  articles  were  built  in  different 
facilities  with  different  processes  and  people.  There  was  no  real  transition 
to  production,  but  rather  a  sudden  shift  firom  development  to  production. 


^Why  Some  Weapon  Systems  Enconnter  Production  Problems  While  Others  Do  Not:  Six  Case  Studies 
CGAO/N8iA.D-85-34,  May  24, 1985). 
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One  of  the  programs  at  that  time,  the  F-16  fighter,  had  a  much  better 
experience.  We  noted  that  development  prototypes  were  built  in  the 
production  facility  using  as  many  production  processes  and  people  as 
possible.  This  made  the  transition  to  production  much  more  gradual  and 
was  one  of  several  factors  that  helped  prepare  the  F-16  for  production. 
Although  the  F-22  is  attempting  to  make  a  bigger  technological  leap  than 
the  F-16  did,  the  development  prototypes  are  being  built  in  the  production 
facility  using  some  production  processes  and  people. 
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The  differences  in  the  practices  employed  by  the  best  commercial  firms 
and  DOD  are  not  necessarily  explainable  by  differences  in  tools,  techniques, 
or  talent.  Rather,  the  differences  in  the  actual  practices  reflect  the 
different  demands  imposed  on  programs  by  the  circumstances  or 
environment  in  which  they  were  managed.  Indeed,  the  way  success  and 
failure  are  defined  for  commercial  and  defense  product  developments 
differs  considerably,  which  creates  a  different  set  of  incentives  and 
different  behaviors  from  the  people  managing  the  programs.  Specific 
practices  take  root  and  are  sustained  because  they  help  a  program 
succeed  in  its  environment.  In  this  sense,  practices  are  adopted  because 
they  work — ^not  because  they  are  textbook  solutions. 

In  general,  the  success  of  a  commercial  product  development  is 
determined  on  the  basis  of  production  items  sold;  similarly,  failure  is 
clearly  defined  as  the  customer  walking  away  and  purchasing  the  product 
of  a  competitor.  This  definition  of  success,  coupled  with  shorter  cycle 
times,  makes  production  concerns  amain  determinant  in  the  initial 
business  case  to  launch  a  commercial  product  development  and  in 
subsequent  decisions  to  manage  risk  and  make  technology  tradeoffs.  This 
environment  encourages  the  early  identification  of  unknowns  with 
realistic  risk  assessments  so  that  risks  are  not  allowed  to  jeopardize 
success.  Strong  incentives,  both  positive  and  negative,  stress  getting  the 
product  development  right. 

DOD  product  developments  are  launched  much  sooner — ^with  greater 
technology,  cost,  and  schedule  imknowns — and  extend  much  longer. 
Production  concerns  do  not  play  as  big  a  role  early  in  these  programs 
because  production  is  so  far  off  and  thus  is  not  as  critical  to  success.  The 
definition  of  success  is  more  complicated  in  dod.  During  most  of  product 
development,  success  is  defined  in  large  part  as  getting  dod  and  the 
Congress  to  fund  the  development  on  an  annual  basis.  Optimistic 
assessments  of  design  performance  and  cost  help  ensure  this  kind  of 
success;  realistic  risk  assessments  for  unknowns  do  not.  By  the  time 
production  nears,  the  risk  of  failure  is  much  reduced  because  a  vested 
customer  is  not  likely  to  walk  away. 
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Commercial  Practices 
Are  Driven  by  the 
Customer’s 
Acceptance  of  the 
Finished  Product 


The  commercial  firms  we  contacted  launch  a  product  development 
program  only  when  a  solid  business  case  can  be  made.  The  business  case 
basically  revolves  around  the  ability  to  produce  a  product  that  will  sell 
well  enough  to  make  an  acceptable  return  on  investment.  The  point  of  sale 
occurs  after  product  development  is  complete;  program  success  is 
determined  in  production  when  the  customer  buys  the  finished  product.  If 
the  firm  has  not  made  a  sound  business  case,  or  has  been  unable  to  deliver 
on  one  or  more  of  the  business  case  factors,  it  faces  a  very  real  prospect  of 
failure.  Production  is  a  dominant  concern  throughout  the  product 
development  process  and  forces  discipline  and  tradeoffs  in  the  design 
process.  This  environment  encourages  realistic  assessments  of  risks  and 
costs;  doing  otherwise  would  threaten  the  business  case  and  invite  failure. 
For  the  same  reasons,  the  environment  places  a  high  value  on  knowledge 
for  making  decisions.  Incentives  favor  identifying  unknowns  early, 
designating  them  an  appropriate  high  risk,  and  aggressively  eliminating 
them.  Practices,  such  as  achieving  statistical  process  control  before 
production,  are  adopted  because  they  help  ensure  success. 


Production  Considerations  For  the  commercial  firms  we  contacted,  file  main  focus  of  a  product 

Figure  Prominently  in  the  development  program  is  to  produce  and  sell  the  right  product  at  the  right 
Program  Launch  Decision  of  a  program  depends  on  the  product's  sales  amd  not  just 

®  on  its  successful  development.  Therefore,  the  business  case  for  launching 

a  program  considers  production  realities  and  builds  in  natural  curbs  to 
overreaching  for  performance,  cost,  or  schedule.  Although  the 
corporations  demand  considerable  proof  in  the  business  plan  that  the 
product  will  meet  market  expectations,  they  then  provide  full 
support — including  funding — ^for  the  plan  to  succeed.  This  provides  a 
program  with  a  very  solid  baseline  from  the  outset  that  continues  to  serve 
as  a  good  decisionmaking  framework  as  the  program  progresses  and 
problems  arise.  The  day  of  reckoning — ^when  program  success  or  failure  is 
realized — comes  clearly  and  swiftly  after  development  when  the  customer 
takes  delivery  of  the  product  and  is  prominent  throughout  the  product 
development  process. 

Boeing  described  the  business  case  for  the  777-200  product  development 
as  a  “money  wheel”  that  must  be  balanced  across  all  of  its  factors.  These 
factors  include  a  market  opportunity,  a  product  whose  technical  features 
can  satisfy  the  market,  available  investment  capital,  a  cycle  time  short 
enough  to  get  the  product  to  market  on  time,  and  a  unit  production  cost 
that  will  yield  an  acceptable  return  on  investment.  Boeing  informed  us  that 
if  any  factor  gets  out  of  line,  either  through  estimating  errors  or  changing 
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conditions,  the  “wheel”  will  not  turn,  and  profitability — and  perhaps 
corporate  reputation — could  be  lost.  The  program  manager  is  judged  by 
these  standards,  unlike  in  dod. 

Before  Boeing  committed  to  development  of  the  777-200,  it  first 
determined  that  a  market  opportunity  existed  for  a  family  of  at  least  10,000 
new,  medium-lift  airplanes  and  that  the  airlines  would  buy  these  airplanes 
at  a  certain  unit  price.  The  company  then  had  to  make  the  business 
case — ^that  it  could  develop  a  product  with  the  right  features,  in  time  to 
meet  the  market  opportunity,  and  at  an  acceptable  unit  cost.  Boeing  did 
not  commit  to  offer  the  new  aircraft  for  sale  until  the  commercial  group’s 
president  presented  solid  evidence,  on  the  basis  of  predecessor  programs, 
testing,  simulations,  and  analysis,  to  the  board  of  directors  that  all  of  the 
business  case  factors  were  achievable.  The  board  then  approved 
investment  of  capital  based  on  its  assessment  of  the  risk  involved  with 
meeting  the  business  case.  Boeing  program  officials  pointed  out,  however, 
that  it  was  not  the  board  meeting  that  provided  the  incentive  to  get  the 
business  case  right.  Rather,  they  said  that  the  commercial  group  president 
knew  that  the  company’s  investment  capital  was  at  stake  on  such  a  major 
product  development.  An  oversold  business  case  would  lead  to  failure  and 
could  have  a  devastating  effect  on  the  company’s  future.  This  had  a  very 
sobering  effect  on  the  people  that  constructed  the  factors  in  the  business 
case  and  curbed  the  tendency  to  overstate  the  case. 

Hughes  presented  a  very  similar  scenario  for  the  corporate  decision  to  go 
forward  with  the  development  of  the  HS-702  satellite.  Hughes  was  very 
comfortable  with  the  level  of  technical  risk  at  the  outset  because  its 
technology  development  process  had  matured  the  required  technologies 
before  inclusion  in  the  program.  In  fact,  the  biggest  risk  in  its  business 
case  was  ensuring  that  the  market  for  the  product  was  profitable.  To 
resolve  this  risk,  Hughes  spent  considerable  time  educating  its  customers 
about  the  benefits  of  the  proposed  program.  Only  after  potential 
customers  were  convinced  did  Hughes  have  a  solid  business  case  for  going 
forward  with  development. 

The  other  companies  we  contacted — Chrysler,  Cummins,  and  Ford — had 
similar  frameworks  established  for  making  decisions  about  product 
development.  Business  realities,  such  as  capital  investment,  cycle  time, 
execution  in  production,  and  the  customer’s  sensitivity  to  purchase  price, 
created  a  high-stakes  environment.  The  decision  to  begin  a  new 
development  program  was  typically  made  at  the  highest  corporate  level. 
However,  the  business  case,  with  all  of  its  competing  factors,  provided  a 
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framework  and  the  incentives  for  program  managers  to  make  practical 
decisions  about  product  development  based  on  facts  and  data.  This 
framework  held  the  achievement  of  each  factor  as  essential  to  success  and 
committed  the  program  managers  to  assess  all  associated  risks 
realistically.  Getting  a  program  approved  that  embodied  false  or  weak 
information  was  viewed  as  a  failure  because  the  customer  might  not  buy 
the  finished  product.  One  commercial  program  manager  informed  us  that 
his  firm  used  to  routinely  underestimate  the  cost  and  schedule  for 
developing  a  new  product  and  overestimate  its  sales  volume  and  profit 
margin.  He  described  this  practice  as  a  “death  cycle”  that  nearly 
bankrupted  the  company. 


Shorter  Cycle  Times  and 
Strong  Incentives  Foster 
Behaviors  That  Keep 
Programs  on  Track 


Once  a  company  decides  to  launch  a  product  development,  strong 
incentives — ^both  positive  and  negative — ^serve  to  keep  the  programs  on 
track.  To  meet  market  demands,  leading  commercial  companies  build 
relatively  short  cycle  times  into  decisions  to  begin  a  product’s 
development.  Boeing’s  777-200  went  to  production  less  than  6  years  after 
development  began,  Hughes’  HS-702  took  about  26  months,  and  Chrysler 
developed  its  new  sport  utility  vehicle  in  less  than  30  months.  These  short 
timeframes  make  the  day  of  reckoning,  in  terms  of  customer  acceptance 
and  return  on  investment,  close  at  hand.  Consequently,  production — on 
time,  at  rate,  at  cost,  and  with  quality — ^looms  as  a  near-term  reality  that 
continues  to  greatly  influence  subsequent  design  and  configuration 
decisions  within  the  framework  of  the  business  case.  The  incentives  that 
operate  in  the  commercial  environment  encourage  program  managers  to 
want  risks  identified  early,  be  intolerant  of  unknowns,  and  not  rely  on 
testing  as  the  main  vehicle  for  discovering  the  performance  characteristics 
of  the  product.  By  protecting  the  business  case  as  the  key  to  success, 
program  managers  are  conservative  in  their  estimates  and  aggressive  in 
risk  reduction.  Ultimately,  preserving  the  business  case  strengthens  the 
ability  to  say  “no”  to  pressures  to  accept  risks  or  unknowns. 

Boeing  believes  a  number  of  factors  serve  as  incentives  to  keep  a  program 
on  treick.  First,  because  the  fate  of  the  program  (and  to  a  large  extent  the 
fate  of  the  company)  is  in  the  hands  of  the  program’s  design/build  teams 
and  their  leaders,  the  incentive  is  for  them  to  convince  themselves  that  the 
business  case  is  solid  rather  than  to  “sell”  program  estimates  to  corporate 
management.  Second,  the  company’s  investment  capital  is  fixed.  Thus, 
development  cost  increases  on  any  one  program  that  can  be  absorbed  are 
limited  because,  at  some  point,  the  overrun  will  hurt  investment  on  other 
programs.  Third,  the  program’s  ultimate  performance  becomes  part  of  a 
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scorecard  on  the  program  manager’s  impact  on  corporate  profitability.  A 
failure  that  hurts  the  company  financially  will  remain  associated  with  the 
program  manager  and  other  people  involved.  Fourth,  if  the  product  does 
not  live  up  to  its  performance  characteristics  (which  include  operation 
and  support  costs),  the  company  will  lose  money  on  product  guarantees. 
Finally,  the  program  management  staff  understand  that  more  is  at  stake 
than  the  program  at  hand;  a  serious  miscue  on  one  program  can  hurt  the 
company’s  reputation  and  damage  other  programs  as  well. 

Although  these  factors  could  be  seen  as  negative  incentives,  Boeing’s 
environment  also  provides  strong  positive  incentives  for  keeping  a 
program  on  track.  The  board  approval  to  launch  a  product  development  is 
a  commitment  to  completely  develop  and  produce  the  item.  There  is  no 
turning  back  unless  something  occurs  that  would  cause  a  serious  decay  in 
the  business  case.  The  investment  funding  is  approved  up  front,  the 
corporation  stands  firmly  behind  the  product  development,  and  the 
program  manager  has  control  over  the  product’s  development  process. 
When  problems  do  occur,  the  corporate  support  does  not  waver,  and 
problem  solving  takes  place  within  a  well-understood 
framework — anchored  in  the  business  case — ^for  making  tradeoffs.  For 
example,  Boeing  realized  early  in  the  777-200  program  that  investment 
costs  would  be  higher  than  anticipated,  largely  due  to  cultural  changes  it 
was  trying  to  accomplish  with  full  use  of  catia  and  design/build  teams. 
Boeing’s  approach  to  resolving  the  problem  began  with  the  premise  that 
the  contractual  commitment  to  deliver  the  aircraft,  with  all  of  its  features, 
on  time,  and  at  the  same  price,  would  be  met.  The  problem  was  then 
limited  to  minimizing  the  increase  in  investment  cost  to  keep  the  business 
case  intact. 

Boeing  said  that  it  is  predisposed  to  identifying  and  solving  problems  early 
and  paying  the  up-front  costs  to  do  it  rather  than  accept  the  production 
cost  consequences  of  the  problems  at  the  end.  Its  decisions  on  the  use  of 
aluminum  lithium  and  the  bulkhead  for  the  777-200  wing  tip  are 
demonstrative  of  behaviors  that  value  minimizing  production  risks  after  a 
program  is  underway.  Although  the  aluminum  lithium  could  have  been 
designed  into  the  aircraft  and  offered  performance  improvements,  it  was 
rejected  because  of  the  unknowns  it  presented  for  material  handling  on 
the  production  line  and  for  long-term  performance  under  operational 
conditions.  Similarly,  Boeing  could  have  taken  the  unneeded  bulkhead  out 
of  the  wing  tip  to  reduce  weight,  but  that  idea  was  rejected  because  the 
redesign  would  have  had  a  ripple  effect  on  drawings  and  production 
preparations. 
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Chrysler  officials  discussed  a  similar  environment  for  managing  the 
product  development  process,  emphasizing  the  need  for  discipline  to 
come  from  within  the  program  itself.  Chrysler  establishes  a  target  unit 
price,  specific  performance  objectives,  and  cycle  time  as  part  of  the 
business  case  for  each  product’s  development.  These  factors  are  strongly 
enforced — ^within  a  fixed-price,  firm  schedule  environment — as  a 
discipline  to  manage  risk  and  facilitate  product  tradeoffs.  The  platform 
teams  operate  under  a  “zero-sum”  basis,  meaning  that  if  the  price  of  one 
component  is  higher  than  expected,  then  savings  must  be  found 
elsewhere.  The  platform  teams  are  totally  empowered  to  make  all 
decisions  concerning  their  product  as  long  as  periodic  reviews  by  product 
assurance  teams  indicate  that  all  of  the  performance  objectives  are  being 
met.  If  the  objectives  are  not  being  met,  the  team  receives  increased 
scrutiny  and  outside  direction.  This  practice  creates  a  strong  incentive  for 
product  managers  and  platform  teams  to  keep  product  risk  to  a  minimum 
by  using  known  product  and  process  technologies  and  discovering 
problems  as  early  as  possible. 

Chrysler  explained  an  important  aspect  of  the  program  manager’s  role  is 
to  say  no  to  things,  such  as  immature  technologies,  that  may  disrupt  the 
product’s  cost,  schedule,  or  performance  targets.  Chrysler  believes  that, 
without  the  “bottom  line”  discipline  from  the  program  manager,  cost 
targets  will  not  work.  Any  facet  of  product  development  that  cannot  be 
shovm  to  be  knowledge,  rather  than  projection,  is  considered  an  unknown 
and  designated  a  high  risk.  Design  and  production  issues  are  thus 
equalized;  a  suspension  component  that  can  improve  vehicle  performance 
and  reduce  complexity  will  be  rejected  if  the  proposing  team  cannot 
provide  proof  that  this  component  can  be  produced  at  the  rate  and  cost 
called  for  by  the  business  case.  Figure  3.1  shows  Chrysler’s  Plymouth 
Prowler. 
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Figure  3.1:  Plymouth  Prowler 


Shortening  product  development  cycles  and  other  incentives  help  Chrysler  managers  identify 
risks  and  say  no  when  necessary. 

Source:  Chrysler  Corportation. 


In  this  environment,  every  team  member  is  considered  to  have  high  risk 
until  each  can  demonstrate  otherwise.  Team  members  are  encouraged  to 
volunteer  information  early  on  risks  or  problems  they  are  having  because 
the  other  team  members  will  step  in  to  help,  particularly  if  they  have 
savings  that  can  offset  price  increases.  The  team  members  know  that  if 
one  of  them  fails,  they  all  fail  because  the  business  case  decays.  They  also 
know  that  if  they  help  someone,  they  increase  the  willingness  of  others  to 
help  them.  By  the  same  token,  the  team  member  that  withholds  problems 
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until  late  in  the  development  is  ostracized  for  putting  the  entire  team  in  a 
position  to  fail. 

Ford  officials  believed  that  the  product  development  environment  at  the 
firm  had  gotten  to  the  point  at  which  people  treated  the  company’s  money 
as  if  it  were  their  own.  In  our  discussions  with  these  officials,  we 
discerned  several  factors  that  converged  to  keep  estimates  intrinsically 
sound  during  the  product  development  process.  These  factors  were: 

(1)  the  vast  amount  of  predecessor  data  from  previous  product 
developments  that  can  be  used  as  a  reality  check  for  a  new  program’s 
estimates;  (2)  the  amount  and  quality  of  the  information  being  generated 
for  the  program  at  hand,  including  the  technique  being  used  to  generate 
data;  and  (3)  the  consequences  of  weak  or  overstated  data  on  the 
program’s  success.  Figure  3.2  shows  the  Ford  Motor  Company’s  Lincoln 
Navigator. 
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Figure  3.2:  Lincoln  Navigator 


Ford  reduced  the  amount  of  unknowns  on  the  new  Lincoln  Navigator  by  limiting  product 
requirements  to  proven  technologies. 

Source:  Ford  Motor  Company. 


The  individual  practices  employed  by  commercial  firms  are  consistent 
with  the  incentives  of  their  environment.  The  high  reliance  on  catia 
shortens  cycle  time  and  reduces  risk  by  generating  knowledge  that 
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previously  had  t»  await  the  building  and  testing  of  physical  articles. 
Insistence  on  having  near  100-percent  release  of  drawings  at  the  cdr  drives 
risks  down  as  the  production  line  is  geared  up.  Achieving  statistical 
process  control  before  production  begins  helps  create  certainty  that 
production  costs,  schedule,  and  quality  will  satisfy  the  customer.  In  short, 
these  practices  are  adopted  because  they  work  by  helping  to  ensure 
success  in  the  commercial  environment. 


DOD  Practices  Reflect 
the  Need  to  Succeed 
in  Funding  and 
Managing  the 
Development  Effort 


DOD  must  make  a  defendable  case  before  launching  a  program.  The 
program’s  merits  generally  relate  to  providing  a  needed  capability  within 
the  limits  of  affordability,  dod  typically  defines  and  launches  a  program 
years  earlier  in  the  process  than  a  commercial  product  development,  and 
thus  the  case  for  the  product  is  made  when  much  less  is  known  about 
technology,  cost,  and  schedule.  In  a  very  real  sense,  the  point  of  sale 
begins  much  earlier  on  a  dod  program  and  continues  throughout 
development  as  the  customer  (dod  and  the  Congress)  pays  for  the  product 
on  an  annual  installment  basis  from  program  launch.  Success,  then,  for 
most  of  the  product  development  cycle,  is  measured  in  terms  of  ability  to 
secure  the  next  installment.  Because  this  approval  must  be  won  every 
year,  it  creates  incentives  to  make  the  program’s  case  look  attractive. 


By  the  time  production  begins,  the  customer  is  deeply  invested  and 
unlikely  to  walk  away.  As  a  result,  success,  in  terms  of  program 
continuance,  is  substantially  ensured  before  end  items  are  produced.  For 
these  reasons,  and  because  production  is  often  many  years  from  the 
launch  decision,  it  is  difficult  for  production  realities  and  concerns  to 
exert  as  much  influence  on  a  dod  product  development.  Instead,  design 
features  and  performance  are  more  dominant.  More  unknowns  are 
accepted  on  a  dod  program,  and  their  attendant  risks  are  often 
understated.  This  combination,  which  can  be  devastating  to  a  business 
case,  can  work  in  a  weapon  development  if  it  helps  the  program  get 
launched  and  survive.  Practices,  such  as  deferring  prove-out  of  key 
processes  until  production  begins,  are  compatible  with  this  definition  of 
success. 


Early  Launches  and 
Long-Term  Programs  Put 
Focus  on  Technology 
Development 


For  a  major  product  development  to  be  launched,  dod’s  acquisition 
guidance  calls  for  a  strong  case  to  be  made  that  includes  a  firm  need, 
superior  product  performance,  affordable  cost,  and  feasible  technology.  A 
need  can  stem  from  several  sources,  including  an  increase  in  threat 
capabilities,  the  obsolescence  of  an  existing  weapon  system,  or  the 
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potential  for  a  new  or  expanded  capability  made  possible  by  technology 
advances.  Traditionally,  needs  for  new  weapons  have  been  generated  by 
individual  branches  within  each  service.  In  recent  years,  dod  has  been 
experimenting  with  other  vehicles  for  determining  needs,  such  as 
advanced  concept  technology  demonstrators  to  evaluate  the  utility  of 
matiure  advanced  technologies.  ‘  Once  a  need  has  been  established,  a 
product  development  vying  for  launch  faces  intense  competition  for  initial 
funding. 

A  key  distinction  between  dod  and  commercial  product  developments  is 
the  timing  of  the  launch.  The  launch  decision  on  a  dod  product 
development  is  often  made  years  before  that  on  a  commercial  product.  In 
DOD,  this  decision  is  made  when  the  program  is  approved  to  enter  the 
program  definition/risk  reduction  phase.  The  knowledge  required  to  make 
the  business  case  to  launch  a  commercial  product  development  is 
generally  not  available  for  a  dod  program  until  well  into  the  emd  phase.  By 
way  of  analogy,  the  F-22  fighter  program,  11  years  after  launch  and  about 
midway  through  the  emd  phase,  may  now  be  at  the  point  that  it  would  be 
ready  for  launch  in  the  commercial  environment.  Thus,  when  a  dod 
product  development  competes  for  launch  funding,  it  is  generally  dealing 
with  far  greater  unknowns  than  its  commercial  counterpart.  Figure  3.3 
shows  an  F-22  fighter  plane. 


^DOD  defines  advanced  concept  technology  demonstrators  as  a  means  of  demonstrating  the  use  of 
mature  technology  to  address  urgent  military  needs.  The  demonstrators  are  not  acquisition  programs, 
but  are  designed  to  provide  a  residual,  usable  capability  upon  completion.  If  the  user  determines  that 
additional  units  are  desired,  the  additional  buys  would  constitute  an  acquisition  program. 
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Figure  3.3:  F-22  Fighter  Piane 


The  F-22,  launched  earlier  than  commercial  products,  needed  significant  advances  in  propuision, 
low-observables,  and  avionics  technologies  to  meet  performance  requirements. 

Source:  DOD. 


Even  though  less  information  about  a  new  dod  product  development  is 
available  at  the  time  of  launch,  the  competition  for  funding  requires 
detailed  projections  to  be  made  from  what  information  does  exist. 
Although  DOD  is  attempting  to  ease  the  technical  requirements  of 
programs,  a  new  product  development  is  encouraged  to  possess 
performance  features  that  distinguish  it  from  other  systems.  For  example, 
the  F-22  will  be  faster  than  the  F-117  and  stealthler  than  the  F-15;  the 
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RAH-66  Comanche  helicopter  will  be  faster,  stealthier,  and  smaller  than 
the  AH-64  Apache.  Consequently,  aspiring  dod  programs  have  incentives  to 
include  performance  features  and  design  characteristics  that  rely  on 
immature  technologies. 

Untempered  by  knowledge  to  the  contrary,  the  risks  associated  with  these 
technologies  are  deemed  acceptable.  We  have  often  reported  on  the 
subsequent  realization  of  such  risks  in  the  form  of  problems  later  in  the 
product  development.  In  a  1993  study  of  seven  weapon  systems,  Rand 
found  that,  despite  policies  to  the  contrary,  the  technology  underlying 
several  programs  was  clearly  not  fully  developed  and  ultimately  caused 
substantial  difficulties.^  Production  realities,  critical  to  matching 
technological  capabilities  with  customer  requirements  on  commercial 
programs,  are  too  far  away  from  the  dod  launch  decision  to  have  the  same 
cinbing  effect  on  technology  decisions.  Indeed,  in  support  of  the  Defense 
Manufacturing  Council,  a  multi-discipline  team  firom  industry  and 
government  pinpointed  a  major  cause  of  acquisition  problems  in  dod  as 
the  imbalance  between  product  goals  and  the  maturity  of  engineering  and 
manufacturing  processes  used  to  reach  those  goals. 

Other  pressures  on  dod  programs  at  launch  make  tough  demands  for 
knowledge  that  does  not  yet  exist.  A  product  development  deemed  worthy 
cannot  be  launched  unless  development  and  production  funding  is 
available  over  the  right  time  period.  The  product’s  development  and 
production  cost,  as  well  as  timing,  must  fall  within  available  funding. 
Because  dod  relies  largely  on  forecasts  of  cost,  schedule,  and  performance 
that  are  comparatively  soft  at  this  stage,  success  in  funding  competition 
encourages  the  cost  and  schedule  estimate  to  be  squeezed  into  profiles  of 
available  funding.  Additional  product  requirements,  such  as  high  reliability 
and  maintainability,  serve  to  make  the  fit  even  tighter. 

Ultimately,  the  demands  of  successfully  competing  for  launch  funds  make 
for  a  much  different  business  case  on  a  dod  product  development. 
Compared  with  commercial  programs,  the  dod  environment  launches 
product  developments  that  embody  more  technical  imknowns  and  less 
knowledge  about  the  performance  and  production  risks  they  entail.  These 
unknowns  place  a  much  greater  focus  on  demonstrating  and  proving  out 
technology  during  the  remainder  of  product  development  than  we  found 
on  commercial  programs.  Equally  important,  the  dod  program  is  launched 
with  customer  funding.  Even  though  the  competition  for  funding  will 
continue  throughout  the  program’s  development,  the  point  of  sale  begins 


^^Barriers  to  Managing  Risk  in  Large  Scale  Weapons  System  Development  Programs^  Rand,  1993. 
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with  program  launch.  When  a  commercial  product  development  is 
launched,  the  customer’s  willingness  to  buy  is  still  undemonstrated  and 
thus  remains  a  significant  risk  and  a  powerful  incentive. 


Incentives  on  DOD 
Programs  Encourage 
Different  Behaviors  From 
Managers 


As  a  product  development  proceeds  in  dod,  its  success  is  still  measured  in 
terms  of  the  funding  it  receives.  Success  translates  into  getting  the  funding 
requested  each  year;  failure  can  mean  anything  from  a  significant  funding 
cut  to  cancellation.  In  a  sense,  the  day  of  reckoning  for  a  dod  product 
development  comes  every  year  with  the  budget  process.  The  most 
important  issues  facing  the  program  at  the  time  will  dominate  the 
attention  of  program  managers  and  decisionmakers.  These  issues  tend  to 
be  related  to  the  successM  demonstration  of  technology.  In  contrast,  the 
day  of  reckoning  for  commercial  programs  is  later  in  the  program  cycle:  at 
the  sale  of  the  produced  item.  The  business  case  with  the  production 
conclusion  sustains  the  commercial  product  development’s  focus  through 
future  decisions  and  problem  solving.  Again,  shorter  cycle  times  are  key  to 
sustaining  this  focus. 


Unlike  commercial  programs,  dod  programs  do  not  receive  full  corporate 
support  throughout  development.  Even  though  many  individual  programs 
compete  for  funding  at  any  given  time,  competition  also  exists  at  the 
aggregate  level,  among  the  services  for  their  portion  of  the  budget.  In 
addition,  dod  programs  face  scrutiny  by  service  executives,  the  Office  of 
the  Secretary  of  Defense,  independent  cost  estimating  and  test  agencies, 
audit  agencies,  and  several  committees  and  subcommittees  of  the 
Congress.  Given  this  amount  of  competition  and  oversight,  the  detection 
of  a  problem  on  an  individual  program  makes  that  program  vulnerable  to 
criticism  and  possible  loss  of  funding  support.  Ironically,  it  is  these  same 
pressures  that  encourage  overreaching  at  the  time  of  program  launch.  This 
situation  contrasts  with  that  of  the  commercial  product  development, 
which  eryoys  full  corporate  support  once  it  is  launched;  in  turn,  the 
program’s  estimates  are  kept  realistic  by  the  knowledge  that  the  program’s 
success  may  determine  the  firm’s  future. 


The  pressures  and  incentives  in  the  dod  environment  explain  why  the 
behaviors  of  managers  and  other  sponsors  of  product  developments  differ 
from  those  in  commercial  programs.  According  to  a  1994  study  done  for 
the  Under  Secretary  of  Defense  for  Acquisition,  government  program 
managers  found  their  formal  role  of  objective  program  management  at 
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odds  with  their  informal  role  of  program  advocates.®  According  to  the 
study; 


“A  feeling  of  responsibility  for  program  advocacy  appears  to  be  the  primary  factor  causing 
government  managers  to  search  aggressively  and  optimistically  for  good  news  relating  to 
their  programs,  and  to  avoid  bad  news,  even  when  it  means  discrediting  conventional 
management  tools  that  forecast  significant  negative  deviations  from  plan.” 

In  this  environment,  risks  in  the  form  of  ambitious  technology 
advancements  and  tight  cost  and  schedule  estimates  are  accepted  as 
necessary  for  a  successful  launch.  Problems  or  indications  that  the 
estimates  are  decaying  do  not  help  sustain  the  program  in  subsequent 
years,  and  thus  their  admission  is  implicitly  discouraged.  An  optimistic 
production  cost  estimate  makes  it  easier  to  launch  a  product  development 
and  sustain  annual  approval;  admission  that  costs  are  likely  to  be  higher 
could  invite  failure.  There  are  few  rewards  for  discovering  and  recognizing 
potential  problems  early  in  the  dod  product  development.  For  commercial 
product  developments,  an  optimistic  production  cost  estimate  will  mean 
failure  of  sales  or  profit;  admission  of  cost  increases  early  invites 
aggressive  problem-solving  behaviors  to  restore  the  business  case.  The 
behavior  of  tolerating  unknowns  and  not  designating  them  the  same  risk 
value  as  in  the  commercial  environment  is  rational  in  the  dod  environment 
because  there  is  little  incentive  to  admit  to  high  risks  before  it  is 
absolutely  necessary,  as  long  as  the  resulting  estimates  are  accepted  by 
DOD  and  the  Congress.  In  fact,  admitting  risk  may  doom  the  program. 

Behaviors  toward  testing  follow  a  similar  logic.  On  commercial  product 
developments,  much  more  is  known  about  the  product’s  performance  at 
the  beginning  of  development.  Testing  is  used  to  confirm  knowledge  and 
identify  weaknesses  or  limits  in  the  product.  It  is  consistent  with  a  firm’s 
anxiety  to  eliminate  unknowns  to  preclude  failure  in  production,  dod 
product  developments  are  much  more  dependent  on  testing  to  discover 
technical  performance  characteristics  and  answer  the  question  of  whether 
the  product  will  work,  dod  tests  serve  more  than  the  purpose  of 
discovering  or  confirming  performance  characteristics — they  are 
examinations  on  which  the  program  must  get  good  grades  or  face  failure 
in  the  form  of  withdrawal  of  support.  Good  test  results  can  help  a 
program,  whereas  negative  test  results  are  equated  with  failure. 
Unknowns,  then,  present  a  safer  course  of  action;  if  testing  does  not  occur 
imtil  late  in  the  product  development,  forecasts  of  product  performance 
will  serve  as  the  only  information  available. 


^Critical  Issues  in  the  Defense  Acquisition  Culture^  Defense  Systems  Management  College — Executive 
Institute,  December  1994. 
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Cycle  time  is  an  important  determinant  of  what  pressures  the  environment 
brings  to  bear  on  a  product  development.  In  a  dod  product  development, 
cycle  times  can  be  10  to  15  years,  whereas  the  tenure  of  a  program 
manager  is  more  likely  to  be  2  to  4  years.  One  senior  commercial  program 
manager  informed  us  that  it  is  unreasonable  to  expect  that  a  program 
manager  can  truly  focus  on  anything  that  is  more  than  3  years  off;  events 
beyond  that  timeframe  are  not  powerful  motivators.  Thus,  it  is  hard  for 
production  success  to  present  the  same  reality  for  the  first  few  managers 
of  a  DOD  program  and  the  decisions  they  are  involved  in.  To  some  extent, 
DOD  may  attempt  to  counteract  the  disparity  between  the  length  of  a 
program  manager’s  tenure  and  that  of  the  program  being  managed  through 
formal  reviews.  Production  readiness  reviews  are  an  example;  they  do  not 
have  a  commercial  counterpart  because  readiness  for  production  is 
intrinsic  to  the  commercial  product  development.  The  production 
readiness  review  does  not  provide  the  same  incentives  for  success 
because  it  is  also  prey  to  the  pressure  to  accept  unknowns  and  assess 
them  as  tolerable  risks,  dod’s  practices,  which  allow  key  production 
knowledge  to  be  gained  concurrent  with  production,  are  a  logical 
consequence  of  the  dod  environment.  If  problems  arise  in  production, 
there  is  not  nearly  the  risk  of  failure  that  a  commercial  product  faces;  by 
the  time  a  weapon  enters  production,  the  point  of  sale  to  the  customer  has 
already  occurred. 

We  did  not  attempt  to  make  a  formal  comparison  between  the  capabilities 
of  DOD  and  commercial  program  managers.  We  believe  that  commercial 
program  managers  have  one  advantage  in  that  they  are  likely  to  have  more 
experience  with  repeated  product  developments  than  a  dod  program 
manager.  Aside  from  that  advantage,  we  did  not  observe  that  commercial 
managers  were  somehow  better  or  more  ethical  than  their  dod 
counterparts.  As  we  previously  reported,^  dod  program  managers  and 
other  sponsors  do  not  act  irrationally  or  with  bad  intentions.  Rather,  they 
see  the  acquisition  of  the  weapons  under  their  purview  as  aligned  with 
national  interests.  They  do  what  they  believe  is  right,  given  the  pressures 
they  face.  The  difference  is  that  the  definition  of  program  success 
determines  what  is  right,  and  success  in  the  dod  environment  is  different 
from  success  in  the  commercial  environment. 


^Weapons  Acquisition:  A  Rare  Opportunity  for  Lasting  Change  (GAO/N8IAD--9?»-15,  Dec.  1992). 
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DOD  has  embarked  on  several  initiatives  that  draw  lessons  from 
commercial  practices,  such  as  cost  as  an  independent  variable  (CAiv), 
integrated  product  teams  (ipt),  use  of  past  performance  data,  and 
performance  specifications,  dod  has  also  recently  set  up  a  funding  reserve 
to  offset  unexpected  cost  growth  to  mitigate  the  effect  of  unknowns  on 
acquisition  programs.  These  initiatives  could  have  a  positive  effect  on  the 
transition  of  weapons  into  production  if  the  environment  for  launching 
programs  and  appraising  risks  can  be  changed  to  provide  the  right 
incentives.  Studies  sponsored  by  dod  and  the  defense  industry  call  for 
changes  that  could  help  shape  such  an  environment.  A 1996  study  by  the 
Defense  Science  Board  Task  Force  on  Defense  Acquisition  Reform  calls 
for  replacing  the  dod  acquisition  process  with  a  process  that  combines  the 
best  elements  of  practices  from  commercial  and  defense-unique  activities. 
A  major  tenet  of  the  recommended  process  is  more  emphasis  on 
incremental  technology  advancement,  coupled  with  much  shorter  product 
development  cycle  times.  These  themes  are  also  echoed  in  a  study  by  a 
National  Center  for  Advanced  Technologies  task  force  comprised  of 
defense  firms. 

Examples  from  past  acquisitions  serve  as  reminders  that  changing  the 
mechanics  of  a  weapon’s  development,  without  changing  aspects  of  its 
environment  that  determine  its  incentives,  may  not  produce  desired 
results,  dod’s  guidance  for  preparing  weapons  for  a  successful  transition 
to  production,  some  of  which  is  10  years  old,  already  has  much  in  common 
with  commercial  best  practices.  Thus,  the  challenge  for  dod  and 
congressional  decisionmakers  may  not  lie  so  much  in  the  “how  to”  aspects 
of  product  development  as  in  creating  the  incentives — or  the  reasons  why 
best  practices  will  work  for  program  managers. 


DOD’s  Risk  Reduction 
Policy  Is  Consistent 
With  Commercial 
Practices 


dod  4245.7-M,  Transition  From  Development  to  Production,  a  risk 
reduction  policy  manual  written  in  1985,  provides  metrics  and  tools  for 
minimizing  design  and  production  risk  during  a  product’s  development.  In 
addition,  dod’s  new  procedures  for  major  defense  acquisition  programs 
contained  in  the  new  5000  series  of  acquisition  policies  emphasize  best 
practices  in  assessing  and  mitigating  risk  during  development.  Figure  4.1 
shows  dod’s  main  policy  guidance  on  preparing  weapons  for  production. 
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Figure  4.1:  DOD  Guidance  on  Preparing  Weapons  for  Production 


DOD  has  provided  ample  guidance  to  program  managers  on  how  to  properly  manage  weapon 
system  development  to  successful  production. 


DOD  4245. 7-M  contains  templates  that  provide  guidance  for  when  and  how 
to  perform  trade  studies,  design  reviews,  and  producibility  reviews.  It 
contains  criteria  for  trade  stupes  that  require  an  evaluation  of  new 
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technology  before  selecting  it  for  a  product’s  development  and  suggests 
that  cost,  producibility,  and  quality  should  be  considered  equally  with 
performance.  The  templates  also  suggest  that  a  cdr  should  be  held  when 
95  percent  of  the  product’s  design  is  complete  in  terms  of  engineering 
plans  and  drawings  and  that  the  design  should  be  frozen  and  released 
directly  after  cdr.  The  templates  also  state  that  cdr  results  and  critical 
drawing  release  schedules  and  approvals  are  essential  data  sources  for 
design  release.  Finally,  the  templates  suggest  that  (1)  the  effect  of  the 
design  on  current  manufacturing  processes  should  be  measured  during  the 
design  process,  (2)  any  new  manufacturing  processes  should  be  “proofed” 
during  development  to  ensure  producibility,  (3)  proofing  simulates  actual 
production  conditions  and  environments,  and  (4)  spc  data  are  considered 
essential  program  data  sources  for  qualifying  the  manufacturing  processes 
during  product  development. 

DOD  revised  its  6000  series  procurement  policies  in  March  1996  with  the 
intent  of  defining  an  acquisition  environment  that  makes  dod  a  smart  and 
responsive  buyer  of  the  best  goods  and  services.  Several  themes  described 
in  the  new  6000  series  documents  resonate  with  conunercial  practices: 

•  The  commercial  marketplace  must  be  researched  to  determine  its 
potential  to  meet  system  performance  requirements  and  results 
documented  in  the  initial  operational  requirements  document. 

•  The  acquisition  process  must  consider  both  performance  requirements 
and  fiscal  constraints  as  embodied  in  CAiv. 

•  Acquisition  of  commercial  processes  and  practices  provides  rapid  and 
affordable  application  of  new  technologies  to  meet  dod  mission  needs. 

•  Future  acquisitions  must  take  into  account  customary  commercial  best 
practices  in  developing  acquisition  strategies  and  contracting 
arrangements. 

•  A  streamlined  management  structure  and  event-driven  management 
process  at  dod  would  emphasize  risk  management  and  affordability  and 
would  explicitly  link  milestone  decisions  to  demonstrated 
accomplishments. 

dod’s  5000  series  documents  emphasize  the  interrelationship  between 
establishing  requirements,  managing  the  development  process,  and 
making  funding  decisions  with  the  objective  of  translating  users’  needs 
into  products  with  affordability  as  a  key  discriminator.  The  policy  also 
emphasizes  the  consideration  of  producibility  early  in  a  product’s 
development.  It  states  that  producibility  is  key  to  managing  risk  and  that 
existing  manufacturing  processes  should  be  capitalized  on  when  possible. 
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It  also  states  that  production  should  not  be  approved  until  the  design  has 
been  stabilized,  manufacturing  processes  have  been  proven,  and  facilities 
and  equipment  are  in  place,  dod  has  emphasized  that  programs  should 
meet  users’  needs  with  mature  technology,  when  possible,  by  using 
demonstrations  of  technology  and  incremental  or  evolutionary  product 
developments,  thereby  taking  advantage  of  existing  information  about  cost 
and  performance. 


DOD  Initiatives  Adapt 
Some  Commercial 
Practices 


dod’s  more  recent  initiatives  are  intended  to  reduce  the  cost  and  duration 
of  major  weapon  system  acquisitions.  In  establishing  the  framework  in 
which  cost  is  considered  an  independent  variable,  dod  is  attempting  to 
make  cost  a  prime  reason  for  trading  off  performance  requirements.  Once 
cost  and  performance  tradeoffs  have  been  made,  caiv  establishes  cost  as 
the  overriding  constraint  for  obtaining  the  needed  military  capability  from 
the  new  system.  For  caiv  to  work  properly,  program  managers  will  need 
sufficient  knowledge  of  technological  capability  and  associated  program 
development  and  production  costs,  along  with  a  sound  mechanism  and 
forum  for  making  tradeoffs.  Whether  tradeoffs  are  made  will  depend  on 
the  incentives  in  place  to  make  them,  lod  points  to  the  jdam  as  an  example 
of  how  effective  caiv  can  be  because  use  of  this  tradeoff  technique 
reduced  expected  unit  production  prices  from  $40,000  to  about  $14,000 
per  unit.  Figure  4.2  shows  the  jdam. 


As  the  program  proceeds  through  development,  the  framework  for  making 
decisions  is  to  be  similar  to  the  zero  sum  approach  an  auto  manufacturer 
follows  after  establishing  a  price  target  for  a  new  vehicle.  Today,  however, 
the  career  incentives  for  program  managers  in  dod  are  not  consistent  with 
those  of  the  auto  manufacturers. 


Page  63 


GAO/NSIAD-98-56  Best  Practices 


Chapter  4 

Creating  the  Right  Environment  Is  Key  to 
the  Success  of  Initiatives  to  Improve 
Weapon  Acquisitions 


Figure  4.2:  Joint  Direct  Attack  Munitions  System 


Use  of  cost  as  an  independent  variable  and  integrated  product  teams  enabled  the  projected  unit 
price  of  the  JDAM  to  be  reduced  from  $40,000  to  $14,000. 


iPTs  are  a  means  to  integrate  the  development  of  a  product  and  its 
manufacturing  processes  by  using  multidisciplinaiy  teams  that  represent  a 
variety  of  product  functions,  such  as  engineering,  manufacturing, 
purchasing,  and  accounting.  Traditionally,  these  functions  have  been 
separate  organizations  that  tended  to  get  involved  sequentially  in  the 
product  development  process.  Through  ipts,  the  functions  are  brought 
together  to  make  tradeoffs  that  will  optimize  the  design,  manufacturing, 
and  supportability  processes  for  a  weapon  system.  They  are  important  to 
making  techniques  such  as  caiv  work,  ipts  have  analogies  in  Boeing’s 
designA)uild  teams  and  Chiysler’s  platform  teams.  Many  defense 
contractors  have  already  adopted  the  ipt  approach.  Several  factors  affect 
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the  success  of  ipts,  including  reasonableness  of  the  program  goals,  the 
knowledge  the  members  bring  to  the  team,  the  authority  the  teams  have  to 
make  tradeoffs  to  achieve  the  goals,  and  the  right  disciplines  represented 
on  the  teams. 

DOD  has  an  initiative  to  use  information  on  the  past  performance  of  a 
contractor  to  ensure  that  dod  has  access  to  a  globally  competitive 
industrial  base  capable  of  supplying  the  best  value  goods  and  services,  dod 
believes  that  past  contractor  performance  information  is  an  important 
strategic  tool  that  when  properly  used,  will  allow  dod  to  evaluate  the  risk 
of  using  poor  or  nonperforming  contractors  as  well  as  the  potential  for 
using  excellent  contractors.  This  information  will  allow  dod  to  make  better 
decisions  during  source  selection.  In  the  commercial  sector,  the  use  of 
past  performance  information  for  suppliers  was  generally  found  to  be 
integral  to  improving  product  developments. 

DOD  has  also  made  the  decision  to  use  performance  specifications  on 
weapon  system  developments.  Previously,  many  military  specifications 
told  defense  contractors  specifically  how  they  were  to  accomplish  certain 
tasks.  Performance  specifications  tell  the  contractor  only  what 
performance  is  desired  of  the  end  product  and  not  how  to  get  that 
performance.  Again,  there  is  a  commercial  analogy  in  the  sense  that 
customers  are  generally  not  concerned  with  how  a  product  does 
something,  such  as  get  better  fuel  economy,  as  long  as  performance  is  as 
advertised. 

DOD  is  tracking  the  effects  of  these  acquisition  reform  initiatives  on  seven 
programs,  designated  as  Defense  Acquisition  Pilot  Programs.  These 
programs  were  afforded  early  statutory  and  regulatory  relief  under  the 
provisions  of  the  Federal  Acquisition  Streamlining  Act  of  1994  to  set  the 
example  for  acquisition  reform,  jdam  is  one  of  the  pilot  programs. 
According  to  dod,  the  successful  application  of  commercial  practices 
enabled  these  programs  to  demonstrate  significant  improvements,  dod 
projects  that  acquisition  programs  that  have  benefitted  from  acquisition 
reform  could  reduce  cycle  time  by  25  percent.  Some  of  dod’s  pilot 
programs  have  significantly  reduced  the  amount  of  military  specifications 
and  standards,  jdam  has  eliminated  all  87  of  the  miUtaiy  specifications  and 
standards  that  were  in  its  baseline  contract  in  favor  of  cortunercial 
practices.  A 1997  dod  report  on  the  pilot  programs  credits  the  use  of  ipts 
for  reducing  program  office  size  and  significantly  decreasing  government 
contract  administrative  hours.  The  same  report  states  that  the 
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improvements  indicated  by  the  pilot  programs  appear  to  be  applicable 
across  dod. 

DOD  has  recently  set  up  a  reserve  of  funds  to  be  used  to  mitigate  the  effect 
of  unforeseen  technical  problems  that  might  threaten  to  upset  an 
acquisition  program’s  schedule.  This  initiative  emerged  from  dod’s 
Quadrennial  Defense  Review,  which  stated  that 


“complex,  technologically  advanced  programs  all  bear  some  risk  of  costing  more  than 
planned.  AVhen  unforeseeable  growth  in  costs  occurs,  offsets  from  other  programs  must  be 
found,  which  in  turn  disrupts  the  overall  modernization  program.  Our  programming 
process  must  provide  sufficient  flexibility  in  the  form  of  program  reserves  to  address  this 
risk.” 

DOD  plans  to  begin  accumulating  the  risk  reduction  fund  in  fiscal  year  2000 
and  expects  it  to  grow  to  about  $1  billion  by  fiscal  year  2003.  It  will  be 
accumulated  through  contributions  from  the  Office  of  the  Secretary  of 
Defense  and  the  services. 

We  have  not  evaluated  this  new  funding  mechanism  or  the  way  dod  plans 
to  implement  it.  Nonetheless,  dod’s  use  of  the  reserve  has  the  potential  for 
communicating  to  program  managers  which  practices  will  be  encouraged 
and  which  ones  will  not.  For  example,  if  the  fund  is  used  primarily  to  pay 
for  problems  that  are  revealed  in  late  development  or  early  production,  the 
fund  could  reinforce  existing  incentives  for  not  dealing  with  risks  until 
they  become  problems.  Conversely,  if  the  fund  is  used  to  resolve  risks 
early  and  preclude  problems,  it  could  encourage  risks  to  be  revealed 
earlier  in  programs. 


Experience  on  Past 
Programs 

Underscores  the  Need 
to  Match  Technique 
With  the  Environment 


A  discussion  of  lessons  learned  or  difficulties  encountered  on  past  weapon 
acquisitions  must  recognize  that  the  dod  acquisition  process  has  produced 
the  best  weaponry  in  the  world.  Nonetheless,  the  experiences  of  some 
we^on  systems  serve  to  point  out  the  importance  of  making  the 
environmental  changes  necessary  for  good  practices  to  work.  These 
techniques  then  will  have  the  chance  to  offer  better  outcomes  consistently 
in  terms  of  cost,  schedule,  and  performance. 


In  the  1980s,  the  Army  attempted  to  acquire  the  Sergeant  York  Air  Defense 
Gun  using  a  strategy  that  included  several  features  similar  to  those  in  the 
Defense  Science  Board’s  recommended  process.  These  features  included  a 
competition  between  two  contractors  to  use  components  based  on 
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previously  demonstrated  technology  and  integrate  them  into  a  weapon 
system,  coupled  with  the  freedom  to  trade  off  flexible  performance 
requirements  to  save  costs  and  shorten  cycle  time.  The  resulting  system 
failed  in  operational  tests  after  it  had  entered  the  4th  year  of  production 
and  was  subsequently  canceled.  The  risks  associated  with  integrating  the 
technologies  had  been  underestimated,  and  the  discovery  process  was 
protracted  into  production.  The  techniques  to  foster  tradeoffs,  reduce 
risks,  and  give  the  competing  contractors  the  freedom  to  design  the 
system  did  not,  by  themselves,  change  the  Sergeant  York’s  environment 
into  one  that  encouraged  the  early  discovery  of  problems. 

Fluctuations  in  funding  cause  perturbations  in  weapon  programs  and  are  a 
complication  that  poses  less  of  a  problem  for  the  commercial  firms  we 
contacted.  However,  stable  funding,  by  itself,  will  not  ensure  program 
success.  In  the  1980s,  baselining  a  program  was  a  technique  aimed  at 
enhancing  program  stability,  whereby  a  program  office  “contracted”  with 
top  management  to  develop  a  system  that  met  basic  performance,  cost, 
and  schedule  requirements  in  exchange  for  stable  funding  and  minimal 
interference.  The  1986  Packard  Commission  cited  the  B-IB  bomber  as  an 
example  of  a  baselined  system.  Despite  eiyoying  stable  funding  and 
congressional  support,  the  B-IB  experienced  significant  performance 
shortfalls  and  continues  to  require  substantial  additional  funding  to 
correct  these  and  other  problems.  Similarly,  the  F-117A  program 
experienced  cost  increases,  schedule  delays,  and  substantial  modifications 
despite  stable  funding,  continuous  congressional  support,  and  streamlined 
management.  In  view  of  the  many  pressures  that  characterized  the 
acquisition  environment  for  these  programs,  stable  funding  and  support 
did  not  prevent  other  problems,  such  as  those  stemming  from  ambitious 
performance  requirements  and  concurrent  schedules. 

When  a  stated  program  goal  is  to  trade  performance  for  cost  and 
operational  support  reductions,  the  acquisition  process  must  encourage 
those  tradeoffs.  This  was  not  the  case  for  the  Army’s  Comanche 
helicopter.  This  program  was  initiated  in  the  early  1980s  by  senior  Army 
management  as  a  family  of  lightweight,  multipurpose  helicopters  whose 
justification  centered  on  practicality  rather  than  the  threat.  The  program 
was  expected  to  replace  a  fleet  of  Vietnam-era  helicopters  with  new 
helicopters  that  would  be  up  to  50  percent  less  expensive  to  operate  and 
support.  Within  these  economical  confines,  the  new  helicopters  were  to 
offer  as  good  a  technical  performance  as  was  possible.  However,  specific 
requirements  were  subsequently  developed  through  a 
performance-oriented  requirements  process.  The  program  emerged  as  a 
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threat-based  program  to  yield  the  next-generation,  high-performance 
helicopter  at  a  cost  significantly  higher  than  that  of  the  Apache,  the  Army’s 
most  advanced  and  costly  helicopter. 

The  lesson  to  be  drawn  from  these  experiences  is  not  that  a  weapon’s 
transition  to  production  cannot  be  improved  through  the  use  of  better 
practices.  Rather,  these  experiences  show  that  attempts  to  improve 
individual  acquisitions  require  more  than  a  new  technique  or  approach; 
they  also  require  the  knowledge  needed  to  make  key  tradeoffs  and  risk 
assessments,  as  well  as  the  recognition  by  decisionmakers  that  such 
information  provided  early  is  critical  to  program  success.  We  believe  that 
the  JDAM  and  aim-9X  have  promise  in  this  regard  because  they  have  utilized 
mostly  existing  technology  in  setting  requirements  and,  when  they  did  not 
use  existing  technology,  have  reduced  risk  through  other  means,  jdam 
represents  a  new  munition  built  with  mostly  commercial  parts,  and  the 
AIM-9X  is  an  upgrade  to  an  existing  missile  system  that  largely  relies  on  its 
predecessor’s  technology.  Moreover,  the  launch  decisions  for  the  jdam  and 
the  AIM-9X  missile  have  encouraged  performance  tradeoffs  to  save  time 
and  money.  Figure  4.3  shows  the  aim-9X  missile. 
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The  challenge  for  these  programs  will  be  to  maintain  their  integrity  during 
subsequent  decisions  as  knowledge— and  problems— are  discovered.  It 
also  remains  to  be  seen  to  what  extent  large  programs,  such  as  new  tank 
or  aircraft  development  programs,  can  apply  these  techniques  in  the 
perhaps  higher  pressure  environment  in  which  they  will  be  managed,  dod’s 
1997  report  on  the  Defense  Acquisition  Pilot  Programs  notes  that,  to  fully 
achieve  the  benefits  of  acquisition  reform,  a  complete  cultural  change 
must  be  affected.  The  report  notes  that  despite  significant  top-level 
management  emphasis  on  changing  the  acquisition  process,  cultural 
resistance  to  various  reform  initiatives  has  been  encountered  on  both 
industry  and  government  teams. 


Recent  Studies 
Promote  Shorter 
Cycle  Times  and 
Other  Environmental 
Changes 


In  May  1996,  the  Defense  Science  Board  completed  a  study  that  concluded 
“force  modernization  at  low  cost  and  with  short  cycles — ^and  with  ability 
to  draw  on  world-class  commercial  firms — ^requires  anew  weapons  R&D 
[research  and  development]  process.”  Some  of  the  features  of  the  process 
recommended  by  the  study  directly  address  key  environmental  differences 
we  noted  between  commercial  and  dod  product  developments. 
Specifically,  the  study  recommended  that  dod’s  process 

aggressively  pursue  high-risk  technology  before  inclusion  in  a  weapon 
research  and  development  program; 

employ  product  solutions  chosen  jointly  by  the  Under  Secretary  of 
Defense  for  Acquisition  and  Technology  and  the  Vice  Chairman  of  the 
Joint  Chiefs  of  Staff; 

maintain  competing  alternatives  for  solutions  throughout  product  design; 
make  use  of  already  developed  next-generation  technology,  and 
concentrate  on  evaluations  of  existing  subsystems  as  the  building  blocks 
for  the  concepts  selected  to  meet  the  need; 

use  flexible  performance  requirements  with  fixed-price  and  firm  schedule 
requirements;  and 

rely  on  competitive  forces  and  price-based  contracting  versus  regulations 
and  cost-based  contracting. 


In  the  opinion  of  the  Defense  Science  Board,  adopting  such  an  acquisition 
process  would  supply  effective  hardware  in  small  quantities,  producible 
and  supportable  at  affordable  cost,  with  cycle  times  reduced  by  one-half. 


In  April  1996,  the  National  Center  for  Advanced  Technologies  also 
proposed  a  change  in  the  dod  weapons  acquisition  process  to  reduce  cycle 
time  by  drawing  on  lessons  learned  from  successful  military  and 


Page  70 


GAO/NSIAD-98-56  Best  Practices 


Chapter  4 

Creating  the  Right  Environment  Is  Key  to 
the  Success  of  Initiatives  to  Improve 
Weapon  Acquisitions 


commercial  programs.  The  center  notes  that  concepts  such  as  integrated 
product/process  development  and  caiv  are  constructive  but  that  previous 
good  ideas  did  not  succeed.  Its  proposal  for  change  calls  for  a  new  culture 
that  relies  on  an  affordable,  incremental  approach  that  could  reduce 
product  development  cycle  times  by  3  to  5  years.  The  new  culture  features 

an  incremental  approach  to  performance,  with  a  threshold  or  minimum 
performance  for  the  initial  battle  group  with  incremental  upgrades  and 
requirements  that  would  be  managed  through  cost  tradeoffs  to  keep 
performance  and  cost  in  balance,  avoid  grand  designs,  and  mitigate  risk. 

The  December  1994  study  by  the  Defense  Systems  Management  College, 
Critical  Issues  in  the  Defense  Acquisition  Culture,  made  several 
recommendations  to  create  an  environment  for  weapon  systems 
development  that  encourages  realism  in  reporting  program  status 
information.  Perhaps  the  most  significant  recommendation  is  that  the 
individual  military  services  transfer  control  of  their  acquisition 
organizations  and  people  to  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology.  The  study  noted  that,  by  doing  so,  the  Under 
Secretary  would  then  be  empowered  to  reward  candor  and  realism  in 
reporting  through  the  use  of  assignments,  transfers,  and  promotions. 
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Conclusions 


Product  development  in  commercial  ventures  is  a  clearly  defined 
undertaking  for  which  firms  insist  on  high  levels  of  knowledge  before 
starting.  Once  underway,  these  firms  demand — and  get — ^specific 
knowledge  about  a  new  product  before  manufacturing  begins.  The  process 
of  discovery— the  accumulation  of  knowledge  and  the  elimination  of  risks 
or  unknowns— is  completed  for  the  best  commercial  programs  well  before 
production  units  are  made.  In  our  analysis,  we  characterized  this 
knowledge  in  terms  of  three  points  or  junctures:  the  match  between 
requirements  and  technology,  the  ability  of  the  design  to  perform  as 
expected,  and  the  ability  to  produce  the  product  on  time  and  at  the  right 
price.  Not  having  this  knowledge  when  demanded  constitutes  risks  that 
the  firms  find  unacceptable.  Immature  or  undeveloped  technology  cannot 
meet  these  demands.  It  is  essentially  kept  out  of  commercial  product 
development  programs  because  of  the  risks  it  poses  to  a  product’s  price 
and  schedule.  Increases  in  price  and  schedule  devalue  a  commercial 
product’s  worth.  Immature  technology  is  managed  separately  until  it  can 
meet  the  demands  for  product  development. 

DOD  does  not  insist  on  the  same  level  of  knowledge  when  it  decides  to 
begin  a  weapon  development  program.  In  the  dod  acquisition  process,  a 
clear  delineation  is  not  made  between  technology  development  and 
product  development.  In  fact,  programs  are  launched  in  the  technology 
development  phase,  and  the  pursuit  of  new  technology — ^the  discovery 
process — continues  through  emd  and  even  into  production.  Not  having  the 
same  level  of  knowledge  as  commercial  firms  explains  much  of  the 
turbulence  in  dod  program  outcomes  as  the  transition  to  production  is 
made.  It  is  a  predictable  consequence  that  can  be  forecast  early  by  the  use 
of  knowledge  points  or  other  metrics,  dod  has  had  guidance  in  place  that 
provides  policies  and  metrics  that  have  much  in  common  with  what 
commercial  firms  demand  in  terms  of  knowledge.  However,  the  way  the 
tools  for  implementing  such  guidance  are  used,  such  as  cdrs  and 
production  readiness  reviews,  understates  the  risks  present.  Ironically, 
commercial  firms  that  have  weeded  out  unknowns  and  minimized  the 
discovery  process  still  identified  some  risks  as  high  that  needed  to  be 
resolved  in  their  product  developments,  dod  programs,  which  accepted 
more  unknowns  and  technical  advances,  did  not  assess  risks  as  high. 

Environmental  differences  underlie  the  differences  in  the  product 
development  practices  we  observed  between  the  dod  programs  and 
commercial  firms.  At  the  core  of  these  differences  are  the  definition  of 
success  and  the  time  span  of  programs.  In  commercial  programs,  success 
is  defined  by  the  sale  of  production  units  to  the  customei^  product 
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development  activities  exist  only  for  production.  Until  the  sale  is  made, 
the  investment  money  of  the  product  developer  is  at  risk.  Product 
development  cycle  times  are  short  enough  that  production  realities  are  a 
major  factor  in  assessing  and  eliminating  risk  in  the  commercial  sector. 
Behaviors,  such  as  candor  about  risk  and  an  intolerance  for  unknowns, 
and  practices,  such  as  ensuring  design  performance  and  demonstration  of 
all  critical  production  processes  by  cdr,  are  encouraged  because  they 
facilitate  success. 

In  DOD,  programs  are  launched  with  immature  technology,  product 
development  cycle  times  are  much  longer,  and  production  realities  figure 
less  prominently  in  early  tradeoff  decisions.  Success  is  defined  more  as  the 
ability  to  secure  annual  funding  installments  from  the  customer  through 
the  budgetary  process  to  sustain  the  development  effort.  Thus,  the  point  of 
sale  essentially  occurs  when  the  program  is  launched.  By  the  time 
production  begins,  the  sale  of  the  product  has,  to  a  large  extent,  already 
occurred,  dod’s  acceptance  of  unknowns  and  practices  that  understate 
risk  are  consistent  with  program  success  as  defined  in  its  environment. 

Commercial  practices  for  gaining  knowledge  and  assessing  risks  can  help 
produce  better  outcomes  on  dod  acquisitions.  For  such  practices  to  work, 
however,  the  knowledge  they  produce  must  help  a  dod  program  succeed 
in  its  environment.  Thus,  the  dod  environment  must  become  conducive  to 
such  practices.  At  least  two  factors  are  critical  to  fostering  such  an 
environment.  First,  program  launch  decisions  must  be  relieved  of  the  need 
to  overpromise  on  performance  and  resource  estimates.  The  pressure  to 
amass  broad  support  at  launch  creates  incentives  for  new  programs  to 
embrace  far  more  technology  development  than  cortunercial  programs. 
The  objectives  of  technology  development,  as  well  as  what  is  demanded  of 
knowledge  and  estimates,  differ  from  those  of  product  development. 
Clearly,  dod  has  to  develop  technology,  particularly  the  technology  that  is 
unique  to  military  applications.  However,  by  separating  technology 
development  from  weapon  programs,  dod  could  insist  on  higher  standards 
for  knowledge  on  its  programs  and  get  better  results  when  those  programs 
transition  to  production. 

Second,  once  a  program  is  underway,  it  must  become  acceptable  to 
identify  unknowns  as  high  risks  so  that  they  can  be  aggressively  worked 
on  earlier  in  development.  Commercial  firms  insist  on  knowledge 
measured  against  a  criterion  for  assessing  risk.  Fixed  prices  and  firm 
schedules  help  form  the  basis  for  such  assessments.  Firms  then  make 
decisions  to  preserve  the  business  case  by  eliminating  risks.  The  result  is 
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discipline  provided  from  within.  In  contrast,  the  identification  of  high  risk 
is  more  likely  to  invite  damage  to  the  funding  case  on  dod  programs. 
Unknowns  are  more  likely  to  be  accepted,  and  knowledge  is  more  likely  to 
be  traded  to  preserve  resource  estimates.  Releasing  drawings  behind 
schedule  indicated  the  attainment  of  less  knowledge  about  design  and 
manufacturing  on  the  F-22  program,  but  production  readiness  reviews  did 
not  identify  any  areas  of  high  risk.  It  took  an  external  team— discipline 
from  outside — ^to  assess  production  risks  as  having  significant  cost 
consequences. 

Decisions  made  by  the  Office  of  the  Secretary  of  Defense  and  the  services 
to  advance  programs  with  significant  unknowns  or  advance  programs  with 
known  high  risks  but  without  sufficient  resources,  send  signals  to 
acquisition  managers  that  current  practices  work  and  are  acceptable. 
These  decisions  define  what  success  means  in  dod  and  what  practices 
contribute  to  success.  The  newly  created  risk  reserve  could  become  one 
vehicle  for  signaling  what  behaviors  and  practices  are  encouraged  on 
individual  programs. 


Recommendations 


To  close  the  gap  between  policy  and  practice,  we  recommend  that  the 
Secretary  of  Defense  take  steps  to  ensure  that  sound  standards  for  the 
timing  and  quality  of  production-related  knowledge  are  applied  to 
individual  weapon  system  programs  and  used  as  a  basis  for  assessing 
production  risks  and  for  making  tradeoffs.  These  standards,  which  can 
already  be  found  to  some  extent  in  existing  dod  guidance,  should  draw 
from  commercial  practices  and  could  include  the  release  of  engineering 
drawings,  identification  of  key  production  processes,  demonstration  of 
risky  or  new  production  processes,  and  achievement  of  statistical  process 
control.  Such  standards  could  enable  manufacturing  representatives  on 
iPTs  to  flag  something  as  high  risk  if  it  would  delay  drawing  release  and  the 
achievement  of  statistical  process  control  beyond  the  standard  of 
acceptability.  Identifying  the  impact  of  such  deficits  in  production 
knowledge  could  help  program  managers  to  say  “no”  to  proposals  to 
accept  unknowns  and  could  force  tradeoffs  in  the  design. 

To  make  the  environment  for  dod  product  developments  more  conducive 
to  the  techniques  used  by  commercial  firms,  we  also  recommend  the 
Secretary  of  Defense  redefine  the  point  for  launching  programs  as  the 
point  at  which  technology  development  ends  and  product  development 
begins.  This  recommendation  is  made  without  prejudice  toward  the 
necessity  of  technology  development  but  rather  with  the  intent  that 
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programs  could  be  better  managed  if  such  development  was  conducted 
outside  their  purview.  Thus,  concomitant  with  defining  the  launch  point 
later  in  the  acquisition  cycle  must  be  the  willingness  of  decisionmakers  in 
DOD  and  the  Congress  to  support  research  and  development  efforts  that 
will  advance  technology  and  establish  the  basis  for  determining  which 
technologies  and  subsystems  have  the  mettle  to  meet  the  performance, 
production,  and  precision  estimating  demands  of  product  development.  If 
extenuating  circumstances  necessitate  including  technology  development 
in  a  program,  this  should  be  recognized  as  a  departure  from  sound 
practices,  accorded  a  high  risk,  and  funded  and  paced  accordingly. 

Finally,  we  recommend  that  on  individual  program  decisions,  the 
Secretary  of  Defense  or  Ws  designee  send  the  signals  that  create 
incentives  for  acquisition  managers  to  identify  unknowns  and  ameliorate 
their  risks  early  in  development.  Incentives  could  take  the  form  of  a 
decision  to  fully  fund  one  program’s  efforts  to  mitigate  a  high  risk 
identified  early  or  requiring  another  program  in  which  risks  are  revealed 
late  to  absorb  the  associated  financial  consequences. 


Matters  for 

Congressional 

Consideration 


Because  of  its  critical  role  in  creating  the  environment  for  what 
constitutes  program  success  and  which  practices  will  work,  the  Congress 
may  wish  to  consider  supporting  the  Secretary  of  Defense’s  efforts  to 
create  such  an  environment  through  changes  to  the  acquisition  process 
that  provide  program  managers  clear  incentives  for  gaining  sufficient 
knowledge  at  key  points  in  weapon  acquisition  programs.  The  best 
commercial  practices  described  in  this  report  suggest  what  may  constitute 
“sufficient”  levels  of  knowledge.  If  dod  takes  steps  to  manage  technology 
development  efforts  outside  the  bounds  of  individual  we^on  system 
programs,  the  Congress  may  wish  to  consider  providing  the  funds  needed 
for  such  efforts.  The  Congress  could  also  help  create  the  right  incentives 
on  individual  programs  by  favorably  considering  dod  funding  requests  to 
mitigate  high  risks  early  in  a  program  and  cautiously  considering  late 
requests  for  funds  to  resolve  problems  that  should  have  been  addressed 
earlier. 


Agency  Comments 
and  Our  Evaluation 


DOD  concurred  with  a  draft  of  this  report  and  all  of  its  recommendations 
(see  app.  I),  dod  noted  that,  although  it  accepts  technical  risks  to  provide 
superior  weapons  that  often  differ  firom  risks  in  commercial  industry,  it 
agreed  that  changes  were  needed  in  its  environment  to  adopt  practices 
that  industry  has  used  to  become  leaner  and  more  flexible,  dod  stated  that 
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it  was  pursuing  such  practices  through  the  Defense  Reform  Initiative,  the 
National  Performance  Review  Reinvention  Center,  Management  Reform 
Memorandums,  and  numerous  acquisition  reform  initiatives. 

DOD  also  highlighted  actions  that  it  believes  are  consistent  with  our 
recommendations.  Regarding  redefining  the  point  for  launching  programs, 
DOD  noted  that  its  emd  phase  was  equivalent  to  the  commercial  product 
launch  and  that  it  was  taking  steps  to  ensure  that  technology  development 
was  separated  firom  this  phase,  dod  stated  that,  to  create  incentives  for 
acquisition  managers  to  identify  unknowns  and  risks  early  in  development, 
it  had  established  goals  for  reducing  program  cycle  time  and  obviating 
cost  growth.  DOD  said  it  was  striving  to  identify  metrics  to  be  used  in 
assessing  program  risks  and  cited  several  existing  sources  for  such 
metrics.  In  response  to  the  matter  that  the  Congress  favorably  consider 
DOD  proposals  for  fully  fimding  technology  development  efforts,  dod 
observed  that  full  funding  can,  in  some  instances,  reduce  the  flexibility 
needed  to  manage  these  risky  efforts,  dod  also  provided  technical 
comments,  which  we  incorporated  where  appropriate. 

As  DOD  takes  actions  to  improve  weapon  system  outcomes  by  better 
identifying  and  eliminating  program  risks  early,  we  underscore  the  need 
for  an  environment  that  creates  incentives  for  such  actions  to  succeed.  It 
was  the  right  environment  that  gave  rise  to  the  excellent  practices  we 
found  in  leading  commercial  firms.  Given  the  competitive  pressures  new 
programs  face,  a  weapon  system  program  that  is  started  during  the 
technology  development  phase  (Program  Definition  and  Risk  Reduction) 
is  encouraged  to  accept  significant  technological  risks,  which  are  not 
necessarily  reflected  in  cost  and  schedule  projections.  When  that  program 
reaches  the  emd  point,  it  is  difficult  to  back  off  of  the  original  projections 
and  put  the  program  on  a  footing  in  which  requirements,  available 
technology,  funding,  and  time  are  matched  to  finish  development  and  start 
production.  Yet,  it  is  from  that  point  on,  despite  the  defense-unique 
technological  risks  previously  accepted,  that  a  weapon  system  program 
should  have  much  in  common  with  a  commercial  program,  including  the 
standards  for  knowledge  that  are  demanded. 

Likewise,  once  a  program  is  underway,  perhaps  the  clearest  way  of 
encomraging  the  right  behaviors — specifically,  program  managers’  and 
other  decisionmakers’  willingness  to  identify  and  deal  with  risks  early — ^is 
through  signals  sent  by  individual  decisions  made  on  individual  programs. 
Absent  such  incentives,  mechanisms  that  already  exist  for  identifying  and 
reporting  risks  have  not  worked.  While  we  support  the  search  for 
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improved  metrics  for  production-related  knowledge,  it  is  the  effective 
application  of  good  metrics  that  is  more  pressing.  Such  metrics  should  be 
applied  to  programs  at  key  junctures  in  a  way  that  will  enforce  standards 
for  knowledge  like  those  we  found  at  commercial  companies.  We  did  not 
intend  that  providing  Ml  funding  for  technology  development  efforts 
should  impede  the  flexibility  needed  to  manage  such  efforts.  We  have 
clarified  the  wording  on  that  matter. 
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THE  UNDER  SECRETARY  OF  DEFENSE 
3010  DEFENSE  PENTAGON 
WASHINGTON.  D  C.  20301-3010 


Mf.  Louis  J.  Rodrigues 
Director.  Defense  Acquisition  Issues 
National  Security  and  International 
Affairs  Division 

U  S.  General  Accounting  Office 
Washington,  D.C.  20548 

Dear  Mr.  Rodrigues: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  General  Accounting  Office 
(GAO)  draft  report,  ‘’WEAPONS  ACQUISITION:  Changes  in  DoD’S  Environmen!  Are  Needed 
to  Apply  Commercial  Practices,”  dated  December  2, 1997  (GAO  Code  707188/OSD  CASE 
1439).  The  Department  concurs  in  the  views  expressed  in  the  GAO  draft  report;  however, 
specific  comments  concerning  your  recommendations  and  suggestions  are  e.nclosed. 

Technical  comments  to  the  .'■eporf  were  separately  provided. 

Tne  fundamental  challenge  in  the  acquisition  of  weapons  systems  is  to  provide 
warfighters  the  capability  to  maintain  combat  superiority  while  minimizing  casualties  which 
drives  DoD  to  develop  and  acquire  technologlcaliy  superior  weapon  systems.  DoD  accepts  the 
associated  technical  risk  and  manages  It  aggressively  which  often  differs  from  that  of 
commercial  Industry. 

We  share  your  view,  however,  that  changes  are  needed  In  DoD’s  environment  In  order 
to  adopt  those  business  practices  that  American  industry  has  successfully  used  to  become 
leaner  and  more  flexible.  The  Department  Is  vigorously  pursuing  the  adoption  of  such  practices 
through  the  Defense  Reform  Initiative,  the  National  Performance  Review  Relnventson  Impact 
Center,  as  well  as.  issuing  various  Management  Reform  Memoranda  to  reengineer  processes, 
and  implementing  numerous  acquisition  reform  initiatives  resulting  in  both  reguiation  and  policy 
changes  addressed  in  the  Acquisition  Deskbook  and  the  Federal  Acquisition  Regulations.  In 
addition,  our  aggressive  communication,  education  and  training  program  is  directed  towards 
accelerating  implementation  of  these  ne’w  initiatives.  Thank  you  for  the  opportunity  to  .review 
and  comment  on  the  draft  report. 

Sincerely, 


Enclosure: 
As  Stated 


S.  Gassier 
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Now  on  pp.  74-75. 


Now  on  pp.  74-75. 


Now  on  pp.  74-75. 


GAO  DRAFT  REPORT  -  DATED  DECEMBER  2,  1997 
(GAO  CASE  707188)  OSD  CASE  1489 

‘WEAPONS  ACQUISITION:  CHANGES  IN  DoD’S  ENVIRONMENT  ARE  NEEDED  TO  APPLY 

COMMERCIAL  PRACTICES" 

DoD  COMMENTS  IN  RESPONSE  TO  THE  RECOMMENDATIONS  AND  MATTERS  FOR 
CONGRESSIONAL  CONSIDERATION 


RECOMMENDATIONS 

•  RECOMMENDATION  1:  The  GAO  recommended  that  the  Secretary  oi  Defense  should 
take  steps  to  ensure  that  sound  standards  for  the  timing  and  quality  of  production-related 
knowledge  are  applied  to  individual  weapon  system  programs  and  used  as  a  basis  for 
assessing  production  risks  and  for  making  tradeoffs,  (p.  10,  p.  79/GAO  Draft  Report) 

DoD  RESPONSE:  CONCUR.  The  Department  is  constantly  striving  to  Identify  metrics  that 
can  be  used  in  assessing  program  risk  of  all  types,  particularly  in  the  area  of  production. 

The  Under  Secretary  of  Defense  (Acquisition  and  Technology)  initiated  an  Enterprise 
Metrics  Program  In  1996  which  led  to  establishing  Year  2000  goals  for  reducing  program 
cycle  time  and  obviating  program  cost  growth.  In  addition,  there  are  numerous  production 
and  manufacturing  courses  available  to  the  acquisition  workfoi'ce  through  the  Defense 
Acquisition  University.  Furthermore,  the  Acquisition  Deskbook  (available  on-line)  outlines 
production- related  key  attributes  that  the  Program  Manager  (PM)  should  look  for  in 
assessing  the  extent  and  effectiveness  of  his/her  program's  efforts  in  designing  for 
prcducibility. 

•  RECOMMENDATION  2:  The  GAO  recommended  that  the  Secretary  of  Defense  redefine 
the  point  for  launching  programs  as  the  point  at  which  technology  development  ends  and 
product  development  begins,  (p.  10,  pp.  79-80/GAO  Draft  Report) 

DoD  RESPONSE:  CONCUR.  The  Engineering  and  Manufacturing  Development  (EMD) 
phase  is  currently  the  Department's  "product  launch”  equivalent.  We  acknowledge  the  need 
to  “reinvestigate”  whether  we  are  stringent  and  precise  m  enforcing  that  technology 
development  ends  before  this  point  In  time.  The  Department  has  taken  steps  and  will 
continue  the  efforts  toward  separation  of  technology  development  and  product  development 
through  Advanced  Concept  Technology  Demonstrations  (ACTD)  and  technology  maturation 
programs.  The  Department's  Cycle  Time  Reduction  Task  Force  is  reviewing  appropriate 
decision  pcirits  for  major  defense  acquisition  programs  as  part  of  a  larger  effort  to 
Investigate  alternative  acquisition  processes  aimed  at  reducing  the  time  to  field  recapitalized 
assets. 


•  RECOMMENDATION  3:  The  GAO  recommended  that  the  Secretary  of  Defense  or  his 
designee,  on  individual  program  decisions,  send  the  signals  that  create  incentives  for 
acquisition  managers  to  identify  unknowns  and  ameliorate  their  risks  early  in  development, 
(p.  10,  p.  80/GAO  Draft  Repo.d) 
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Now  on  pp.  74-75. 


Now  on  pp,  74-75. 


Now  on  pp.  74-75. 


DoD  RESPONSE:  CONCUR.  As  a  National  Performance  Reviev.'  Reinventicn  Impact 
Center.  DoD  has  established  Year  2000  acquisition  goals  for  reducing  program  cycle  time 
and  obviating  program  cost  growth. 

MATTERS  FOR  CONGRESSIONAL  CONSiDER.ATiON 

•  SUGGESTION  1:  The  GAO  suggested  that  the  Congress  should  support  the  Secretary 
of  Defense's  efforts  to  create  an  environment  for  program  success  through  changes  to 
the  acquisition  process  that  provide  program  managers  dear  incentives  for  gaining 
sufliclent  knowledge  at  key  points  in  weapon  acquisition  programs,  (pp.  10-11,  p. 
80/QAO  Draft  Report) 

DOD  RESPONSE:  CONCUR. 

•  SUGGESTION  2:  The  GAO  suggested  that  the  Congress  could  help  foster  the  right 
incentives  by  favorably  considering  DoD  proposals  to  fully  fund  technology  development 
efforts  as  v/ell  as  mitigate  high  risks  early  in  product  development  efforts,  (pp.  10-11,  p. 
80/GAO  Draft  Report) 

DOD  RESPONSE:  CONCUR.  During  technology  development  however,  some 
flexibility  is  also  absolutely  critical.  Full  funding  can  potentially  straitjacket  the 
technology  development  process  by  channeling  dollars  to  those  efforts  which  possess 
strong  advocacy  and  ‘clear  promise.*  thus  shifting  money  away  from  less  glamorous 
activities.  If  the  promise  Is  not  borne  out  which  happens  often  In  this  risky  enterprise,  it 
should  be  possible  to  easily  shift  funds  to  other  activities.  Full  funding,  alone  at  this 
stage,  will  produce  a  sunk  cost  mindset  and  limit  our  options.  What  Is  also  needed  is  a 
stronger  linkage  between  the  warfighting  and  development  communities.  One  proposal 
is  to  build  upon  the  successes  of  the  Advanced  Concept  Technology  Demonstration 
(ACTD)  progranr^.  by  initiating  a  number  of  'Demonstrations/  collaborative  warfighter- 
developer  efforts  which  culminate  in  an  operational  demonstration  of  a  specific 
capability.  The  warfighter's  operational  assessment  would  determine  a  need  to  acquire. 
This  would  necessitate  a  shift  away  from  the  serial  development  of  requirement  and 
system,  towards  a  ‘convergence’  phase  in  which  user  and  acquirer  iterate  on  a  particular 
need  statement  and  its  requirement.  Focused  demonstrations  would  (1)  provide  a 
target  for  the  S&T  community  to  'shoot  at/  and,  perhaps  more  Importantly.  (2)  procure 
an  operationally  useful  and  technically  feasible  system  which  could,  if  desired,  be 
acquired  on  a  larger  scale  following  a  milestone  decision. 

•  SUGGESTION  3:  The  GAO  suggested  that  the  Congress  should  carefully  consider 
requests  for  additional  funds  to  resolve  problems  that  should  have  been  addressed 
earlier  in  weapon  acquisition  programs,  (pp.  10-11,  p.  80/GAO  Draft  Report) 

POD  RESPONSE:  CONCUR. 
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